This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Environment Canada’s Invasive Alien Species Partnership Program

October 2009

Report Clearance Steps

Planning phase completed
November 2008
Report sent for management response
September 2009
Management response received
October 2009
Report completed
October 2009
Report approved by Departmental Evaluation Committee (DEC)
October 2009

abbrs used in the report

AAFC
Agri-Foods and Agriculture Canada
ADM
Assistant Deputy Minister
CFIA
Canadian Food Inspection Agency
CITES
Convention on International Trade in Endangered Species of Wild Fauna and Flora
DFO
Fisheries and Oceans Canada
DPR
Departmental Performance Report
ESB
Environmental Stewardship Branch
IAS
Invasive Alien Species
IASPP
Invasive Alien Species Partnership Program
IASSC
Invasive Alien Species Strategy for Canada
IPPC
International Plant Protection Convention
NRCan
Natural Resources Canada
OIE
World Organization for Animal Health
RMAF
Results-based Management and Accountability Framework
RPP
Report on Plans and Priorities
SARA
Species at Risk Act
TRC
Technical Review Committee
WAPPRIITA
Wild Animal and Plant Protection and Regulation of International and Interprovincial Trade Act

Acknowledgments

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee as well as all interviewees and survey respondents who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was led by Robert Tkaczyk, under the direction of the Environment Canada Evaluation Director, Shelley Borys, and included William Blois and Janet King.

This evaluation report was prepared by TDV Global Inc. and the Evaluation Division, Audit and Evaluation Branch.

Table of Contents

Executive Summary

Background

Environment Canada’s Evaluation Division, Audit and Evaluation Branch, conducted an evaluation of Environment Canada’s Invasive Alien Species Partnership Program (IASPP) in the 2008-2009 fiscal year. The evaluation was identified as part of the Departmental Audit and Evaluation Plan approved in April 2008 and was undertaken to fulfil Treasury Board evaluation requirements for program renewal as the program’s terms and conditions are expiring at the end of 2009-2010. The evaluation was initiated in October 2008 and completed in September 2009.

The IASPP is one part of a larger Government of Canada initiative on invasive alien species (IAS). Administered by Environment Canada, the IASPP is a contribution program with a budget of $5 million over five years (from fiscal year 2005-2006 to 2009-2010) designed to reduce the risk of future introductions and to help manage the spread of existing IAS within Canada.

The evaluation examined the IASPP both within the context of EC priorities, as well as within the context of the objectives of the joint national Invasive Alien Species Strategy for Canada (IASSC), of which the IASPP is a component. The scope of this evaluation was limited to examining EC’s participation in the IASSC through the IASPP. The evaluation focused on the time frame of the fiscal years 2005-2006 to 2008-2009. The main objectives of the evaluation were to provide conclusions and recommendations to be applied in the context of program renewal.

Findings Summary

Relevance

Although there is a continued need for a contribution program in Canada to address IAS at the grassroots level, there remain questions with respect to the effectiveness of the program to achieve its overall objectives given issues related to a lack of concise and focused program priorities and limited program funding. Some interviewees felt that diverting the funding to other programs under the national strategy could have a greater impact on addressing IAS-related concerns nationally.

Furthermore, although the IASPP was well-aligned with government-wide priorities in 2005, when the program was created, as of the present, there is only an indirect link between IASPP objectives and current government-wide economic priorities.

The IASPP does contribute to the national strategy, and complements other programs under the national strategy, in that it engages stakeholders beyond the federal and provincial governments, targeting community-level and grassroots organizations.

Environment Canada is perceived as the best situated department to administer the IASPP and coordinate the national strategy, given that it is seen as a neutral party without a vested interest with respect to IAS.

Success

The program is perceived to be contributing to some extent to resolving the challenges related to IAS in Canada. However, given the present limited funding of the program these contributions have resulted in marginal gains at the local/community level, not major impacts at the national level. Furthermore, although there have been marginal gains at the local/community level in raising the awareness of Canadians through training and outreach, as well as through the generation and dissemination of products related to IAS, these do not necessarily indicate that Canadians have become better informed or more engaged in activities on priority IAS issues as a result of the program.

Finally, certain design and delivery limitations, such as uncertainty about the accuracy of available performance data, further limit the possibility of determining the level of success of the program in achieving its expected outcomes and overall objectives.

Cost-Effectiveness

Given the uncertainty about the accuracy of available performance data for the IASPP, a cost-effectiveness analysis could not be conducted. Consequently, cost-effectiveness was measured indirectly by examining program efficiency and key stakeholders’ perceptions of the program’s efficiency.

Evidence collected indicates that the program is being delivered in a cost-efficient manner through its low administrative costs (13.4%), as well as its significant leveraging of matching funds ($1.19:1 ratio).

No alternative approaches to a contribution program were identified that could be more cost-effective or efficient at achieving the same objectives as the IASPP, although some areas of improvement to the present approach were suggested, including the need to focus more on an action-oriented approach to addressing IAS in Canada, as opposed to focusing on education and outreach through the dissemination of information products and training.

It was felt by most interviewees that limited operational funding had a negative impact on the overall effectiveness of the program to achieve its objectives, in that it affected both the management of IASPP contribution agreements, as well as Environment Canada’s ability to fulfil its coordinating role with respect to the national strategy.

Design and Delivery

Overall, the program is being delivered as designed and continues to make operational improvements (e.g., creation of a website, addition of an online application process).

There are a number of areas that need to be addressed in the delivery of the program, including the implementation of an effective performance measurement strategy, communications to proponents related to the overall objectives of the program, and timeliness of funding to successful proponents.

Recommendations

These recommendations are directed to the ADM, ESB.

Recommendations 1 and 2:

Overall, although there is a continued need for a contribution program in Canada to address IAS at the grassroots level, interviewees questioned the effectiveness of the IASPP to achieve its overall objectives given issues related to a lack of concise and focused program priorities and limited program funding. Most felt that although the individual projects funded by the program were generally successful in achieving their respective objectives, these results were at the local/community level. It was therefore felt that the contributions of the program resulted in marginal gains at the local/community level and not major impacts towards addressing IAS-related concerns at the national level. Some interviewees felt that given these results, the limited funds could be diverted to other programs under the national strategy to provide a greater impact at addressing IAS-related concerns at the national level.

Interviewees spoke of the need to re-examine the overall objectives of the program, in order to determine a few key priorities for the program where it could be expected to make a difference. At present, it was felt that the current objectives were too broad, resulting in the program being spread too thinly across the country, providing nominal funding to small localized projects, instead of focusing on a few key priorities that would result in the program making a bigger impact at addressing IAS-related concerns nationally.

Furthermore, limited program funding also had a negative impact on the overall effectiveness of the program to achieve its objectives, as it affected both the management of IASPP contribution agreements and Environment Canada’s ability to fulfil its coordinating role with respect to the national strategy. Many identified the need for a more effective coordination of the national strategy, something they felt was lacking at present.

Recommendation 1:

In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise with counterparts in participating departments and agencies to determine the appropriate role for the IASPP within the context of both the IASSC and EC departmental priorities, along with commensurate funding for the IASPP.

Recommendation 2:

Based on the decision taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, it is recommended that the program’s objectives be re-examined with an aim to develop a concise and more focused set of key priorities for the program to ensure that the IASPP contributes in an effective manner to both the national strategy and EC departmental priorities.

Recommendation 3:

Evidence collected as part of this evaluation illustrated a lack of clear understanding by project proponents about the overall objectives of the IASPP, thereby making it difficult for them to develop projects that link more directly with the program’s objectives. The application guidelines do provide some guidance, but it is at a very high level, with more emphasis on the evaluation criteria and listing examples of the types of activities that are funded, than specifically stating the objectives or the expected results of the IASPP.

Another key weakness of the IASPP lies in its limited ability to demonstrate achievement of its intended outcomes. This is due in part to the absence of a formal mechanism to validate the performance data that are collected by project proponents and provided to the program at the completion of individual projects. Thus, there is uncertainty about the accuracy of the performance data provided. Furthermore, the program presently collects information on approximately 114 key performance indicators, used to measure the level of achievement related to 56 project-level outputs and outcomes. The sheer number of performance indicators, as well as their diverse nature (i.e., environmental, economic, community capacity, outreach, innovation, engaging Canadians, training) has made it difficult for the program to consolidate performance data and use this data to report on the overall success of the program in achieving its outcomes. Furthermore, in some cases, there are issues with respect to the link between these performance indicators and the project-level outputs and outcomes that they are intended to measure. Thus, the program cannot use the indicators to determine whether IASPP-funded projects have been successful in achieving specific project results and overall program objectives.

As a result, it is quite difficult to determine the extent to which the program has been successful in achieving its immediate and intermediate outcomes, as identified in the IASPP logic model.

Based on the decision taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, as well as the articulation of program priorities as per Recommendation 2, it is recommended that:

a. the program clearly communicates its key priorities and objectives to potential project proponents to enable their understanding and thus ensure that projects which are developed are more directly linked to program objectives.

b. the program’s current performance measurement strategy be re-examined with the objective to develop and implement a set of intended project-level outcomes and performance indicators that would allow the program to be better able to demonstrate its results.

Management Response

The Assistant Deputy Minister, Environmental Stewardship Branch, Environment Canada (ADM, ESB) agrees with these three recommendations. Should there be a government decision to renew the IASPP, the program commits to the following actions in response to the three recommendations.

Recommendation 1:

In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise with counterparts in participating departments and agencies to determine the appropriate role for the IASPP within the context of both the IASSC and EC departmental priorities, along with commensurate funding for the IASPP.

If renewed, the specific role of and priorities for the IASPP should focus on the IASCC and EC departmental priorities, the priorities of the Canadian Food Inspection Agency (CFIA) and Fisheries and Oceans Canada (DFO) with whom EC shares governance of the IASPP, and the advice of other involved federal departments and agencies. The following action will be taken by the ADM, ESB to ensure this recommendation is addressed:

December 2009

  • ADM, ESB to discuss role of, priorities and commensurate funding for the IASPP in implementing the renewed IASSC with other federal departments and agencies.

Recommendation 2:

Based on the decision to be taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, it is recommended that the program’s objectives be re-examined with an aim to develop a concise and more focused set of key priorities for the program to ensure that the IASPP contributes in an effective manner to both the national strategy and EC departmental priorities.

Following confirmation of the role and commensurate funding for the renewed IASPP, the program’s objectives will be re-examined to create a more focused set of key priorities aligned with both the national strategy and EC departmental priorities. In doing so, EC will consult with the Canadian Food Inspection Agency, the Department of Fisheries and Oceans and other key involved federal departments and agencies with respect to their priorities for IASPP.

Specific actions to be taken to address this recommendation include:

December 2009

  • Subject to renewal of the IASPP, identify strategic outcomes, objectives and key priorities for the renewed IASPP in consultation with other federal departments and agencies.
  • Program to return to Environmental Sustainability Board with recommendations for future implementation of the IASPP.

Recommendation 3:

Based on the decision to be taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, as well as the articulation of program priorities as per Recommendation 2, it is recommended that:

a. the program clearly communicates its key priorities and objectives to potential project proponents to enable their understanding and thus ensure that projects which are developed are more directly linked to program objectives.

Subject to a renewed IASPP, EC will refine and update the IASPP application guidelines including project selection criteria to ensure greater clarity about the overall objectives of the IASPP, thereby ensuring projects link directly with the Program’s objectives.

Specific actions to be taken to address this recommendation include:

January 2010

  • Refine the application guidelines in line with revised Program objectives.
  • Provide guidance such as a list of eligible and ineligible projects to project proponents to ensure clarity with respect to IASPP objectives and priorities.

b. the program’s current performance measurement strategy be re-examined with the objective to develop and implement a set of intended project-level outcomes and performance indicators that would allow the program to be better able to demonstrate its results.

The performance measurement strategy of a renewed IASPP will be revised in line with the objectives and priorities to be established for the Program. Performance indicators will be streamlined to ensure collection and consolidation of meaningful data which can be used to report on the overall success of the program in achieving its outcomes.

To meet this goal, the following specific actions will be taken.

March 2010

  • Refine current logic model to reflect the new objectives (outcomes) for the Program
  • Complete a Performance Measurement Strategy with the aim to develop performance indicators most relevant to program objectives

March 2011

  • Improve the reporting format, implement an appropriate reporting tool and make other program adjustments as determined by the departmental G&C risk assessment strategy developed as part of the Departmental Action Plan for G&C Reform (CAPE Optimization Initiative)

March 2012

  • Adopt the departmental online application and information management system, which will enhance client service and facilitate application, monitoring and reporting processes for both clients and program staff (CAPE Optimization Initiative)

Contact persons: Bob McLean, Elizabeth Roberts

1.0 Introduction

Environment Canada’s Evaluation Division, Audit and Evaluation Branch, conducted an evaluation of Environment Canada’s Invasive Alien Species Partnership Program (IASPP) in the 2008-2009 fiscal year. The evaluation was identified as part of the Departmental Audit and Evaluation Plan approved in April 2008 and was undertaken to fulfil Treasury Board evaluation requirements for program renewal (as the program’s terms and conditions expire at the end of 2009-2010).

This document presents the findings and recommendations of the evaluation and is organized in the following way. Section 2 provides background information on the IASPP. Section 3 presents the purpose of the evaluation and the methods used to conduct the evaluation. Section 4 presents the evaluation’s findings. Sections 5 and 6 lay out, respectively, the conclusions and recommendations. The management response is found in Section 7.

2.0 Background

Invasive Alien Species (IAS) represent the second greatest threat to biodiversity, after habitat loss. Alien species are species of plants, animals (including fish), and micro-organisms introduced by human action outside their natural past or present distribution. They are also known as exotics or specified as being foreign or non-native. Introductions of alien species may be deliberate or accidental and may be beneficial, as in the examples of corn, wheat, and domestic livestock, or damaging, such as in the cases of leafy spurge, zebra mussels and wild boars.

IAS are those harmful alien species whose introduction or spread threatens the environment, the economy, or society, including human health. Alien bacteria, viruses, fungi, aquatic and terrestrial plants, mammals, birds, reptiles, amphibians, fish, and invertebrates (including insects and molluscs) can all become invaders. In their new ecosystems, IAS become predators, competitors, parasites and diseases of native and domesticated plants, animals and marine life. The impact of IAS on native ecosystems, habitats and species is severe and often irreversible, and can cost billions of dollars each year.1

2.1 Profile

In September 2001, federal, provincial and territorial Ministers responsible for forests, fisheries and aquaculture, wildlife, and endangered species chose IAS as one of four priorities of the Canadian Biodiversity Strategy. With this direction, the federal government, led by Environment Canada, along with the provinces, territories, industry, and non-government groups developed a joint national Invasive Alien Species Strategy for Canada (IASSC), which was approved in September 2004. At this time, direction was given to the federal-provincial-territorial working groups to develop associated implementation strategies for aquatic invasive species, invasive alien plants and plant pests, and terrestrial animals.

The purpose of the IASSC is to minimize the risk of IAS to the environment, economy, and society through a hierarchical approach that prioritizes prevention, early detection, rapid response and management. Specifically, the IASSC aims to prevent new invasions; detect and respond rapidly to new IAS; and manage established and spreading IAS through eradication, containment, and control. The strategy also includes measures to help prevent introductions of IAS from other countries or from species which have moved from one ecosystem to another within Canada. Similar to other countries, the focus of Canada’s IAS policy and management framework is on priority pathways2 of unintentional and intentional introductions. The strategy is applicable to a wide range of sectors, including agriculture, fisheries and aquaculture, wildlife, forests, transportation, industry, and human health.

In October 2005, the implementation strategies developed by the working groups were approved by federal, provincial and territorial Ministers. These strategies focused on risk analysis, science, legislation and regulation, education and outreach, international cooperation, and priority setting and included: The Canadian Action Plan to Address the Threat of Aquatic Invasive Species; The Action Plan for Invasive Terrestrial Plants and Plant Pests; and Canada’s National Wildlife Disease Strategy.

In the 2005 Federal Budget, four departments/agencies received funding of $85M in new resources to implement the IASSC: Natural Resources Canada (Canadian Forest Service) received $10M over five years; Fisheries and Oceans Canada received $20M over five years; Agriculture and Agri-Food Canada (Canadian Food Inspection Agency – Plant Health Division) received $50M over five years; and Environment Canada received $5M over five years for the Invasive Alien Species Partnership Program (IASPP).

2.1.1 Program Description

Environment Canada’s IASPP is one part of a larger Government of Canada initiative on IAS. Administered by Environment Canada, the IASPP is a contribution program ($5 million over five years) designed to reduce the risk of future introductions and to help manage the spread of existing IAS within Canada, by funding initiatives intended to address priority pathways of invasion for invasive alien plants, plant pests and aquatic invasive species. Funded activities encourage stakeholders to prevent the introduction of invasive species by increasing their understanding and awareness of invasive species issues and facilitating responsible decision-making that minimizes the risk of unwanted introductions.

The IASPP is administered through an allotment held in Environment Canada’s reference levels. Funding is used to establish partnerships with the provinces and territories, as well as with industry and non-governmental organizations to promote compliance with regulatory approaches and establish voluntary initiatives for pathways not amenable to regulation.

2.1.2 Expected Project Results

The activities funded through the IASPP are directly linked to the education and outreach component of the IASSC, specifically through the “Plan for Engaging Canadians and Sectors” output in the national IASSC Logic Model (presented in Annex A). Activities are intended to promote voluntary approaches and complement regulatory compliance activities.

Expected results of approved projects fall into three main areas:

  • Active engagement of Canadians in minimizing the risk of invasive alien species;
  • Development of products and tools to reduce unintentional introductions through specific pathways of invasion; and
  • Development of products and tools to increase general public awareness and understanding of invasive species issues.

2.1.3 IASPP Governance Structure

IASPP is overseen by a Steering Committee, which is comprised of the Deputy Minister of Environment Canada, the President of the Canadian Food Inspection Agency, and the Deputy Minister of Fisheries and Oceans Canada, or their designates. Roles and responsibilities of the Steering Committee, defined in the Interdepartmental Memorandum of Understanding between Environment Canada, Canadian Food Inspection Agency and Fisheries and Oceans Canada, include:

  • Approve projects for funding to be released by Environment Canada and, subject to Treasury Board and Finance approval, recommend the amount of funding to be reprofiled between fiscal years;
  • Review annual results to assess progress on outputs and outcomes;
  • Review annual cash flow forecasts and expenditures to date;
  • Report on results and achievements; and
  • Approve changes to the objectives of the Interdepartmental Memorandum of Understanding between Environment Canada, Canadian Food Inspection Agency and Fisheries and Oceans Canada, as appropriate.

An interdepartmental Director General Committee was established to coordinate the overall implementation of the national IASSC including the IASPP.

A Technical Review Committee, coordinated by Environment Canada, was established to provide technical, scientific, and policy advice for the review, selection, and management of projects for the IASPP. The committee is comprised of experts from the three signatory departments and, when appropriate, may include representatives of Natural Resources Canada (Canadian Forest Service); Agriculture and Agri-Food Canada, Parks Canada Agency, and Canada Border Services Agency in an advisory role. The Terms of Reference of the committee were also established in the Interdepartmental Memorandum of Understanding between Environment Canada, Canadian Food Inspection Agency and Fisheries and Oceans Canada.

The activities of the Technical Committee include:

  • Review project proposals and develop recommendations annually for the selection of projects;
  • Develop an annual costed workplan with recommended projects, milestones, targets, and indicators;
  • Submit the proposed workplan with recommendations to the Steering Committee;
  • Monitor annual results to assess progress on outputs and outcomes;
  • Monitor annual cash flow forecasts and expenditures to date; and
  • Report annually on results and achievements to the Steering Committee.

2.1.4 Resource Allocation

As part of the 2005 Federal Budget, the IASPP received a total of $5 million for the period FY 2005-2006 until 2009-2010 ($1 million per year) for the contribution component of the national IASSC. No specific funding for program administration was allocated to the program (i.e., Operations & Maintenance, salary); all operating expenses for the IASPP were covered by Environment Canada’s A-base budget.

2.1.5 IASPP Logic Model

The 2008 Results-based Management and Accountability Framework (RMAF) includes the logic model of the IASPP (Figure 1), which identifies the linkages between the program’s activities and intended outcomes.

The key activities of the IASPP are two-fold: program-related and project-related. These include planning and coordination, project assessment and review, as well as managing agreements (i.e., monitoring progress, reporting on results). Key outputs of the program identified in the RMAF (though not specifically in the logic model) include the following:

  • Contribution agreements with project proponents who undertake activities related to the prevention, early detection, rapid response and management of IAS;
  • Communications products relating to both aquatic and terrestrial IAS developed and disseminated for national use;
  • Education modules developed and utilized where possible and appropriate to raise awareness regarding IAS; and
  • Other key products (e.g., codes of conduct, consumer information pamphlets, best practices) developed in partnership with key stakeholders to minimize the risk of introduction of IAS through priority pathways.

Figure 1: Invasive Alien Species Partnership Program Logic Model

Figure 1: Invasive Alien Species Partnership Program Logic Model

Click to enlarge

3.0 Evaluation Design3

The following sections outline the evaluation purpose and scope, along with the data collection approach and methods used.

3.1 Purpose and Scope

The evaluation examined the IASPP both within the context of EC priorities, as well as within the context of the objectives of the joint national IASSC. The main objectives of the evaluation were to provide conclusions and recommendations to be applied in the context of program renewal. The scope of this evaluation was limited to examining EC’s participation in the IASSC through the IASPP. The evaluation focused on the time frame of the fiscal years 2005-2006 to 2008-2009.

The following evaluation issues were addressed as part of this evaluation:

Relevance

Q1: Does the IASPP continue to be consistent with government-wide priorities and the departmental mandate?

Q2: Is there a continued need for the IASPP?

Success

Q3: Are planned activities being implemented and producing expected outputs?4

Q4: Are immediate and intermediate outcomes being achieved as a result of the program?

Q5: Have there been any unexpected outcomes (positive or negative)? Were any actions taken as a result of these?

Cost-effectiveness/Alternatives

Q6: Are the most appropriate and efficient means being used to achieve objectives relative to alternative design and delivery approaches?

Design and Delivery

Q7: Are program deliverables and expected outcomes identified clearly, and is the program being delivered as designed?

Q8: Are activities and outputs linked to shared outcomes and the overall government-wide objectives?

Q9: Are performance data collected against program activities/outcomes and used for informed decision making?

Q10: Are there appropriate governance structures in place to meet the objectives?

Q11: Has there been an assessment and strategic use of lessons learned?

The evaluation matrix, mapping each question to the related indicators, data sources and methodologies, is presented in Annex B.

3.2 Evaluation Approach and Methodology

This section describes the methods that were used to conduct the evaluation of the IASPP as well as the limitations of the evaluation.

3.2.1 Methods

Document Review

Existing documentation was reviewed early in the data collection phase in order to establish a sound basis for the interviews and survey. Performance monitoring information on specific performance indicators related to program outputs and results was collected and analyzed. The performance measurement system as a whole was also assessed. Gaps, indicator weaknesses and/or data vulnerabilities were identified. Please see Annex C for the list of documentation reviewed.

The goal in this step was to answer as many of the questions within the evaluation matrix as possible and then to validate the findings where necessary and appropriate through other data collection methods.

This data collection method addressed evaluation questions 1 through 11.

File Review

A total of 8 out of 71 completed project files were reviewed at the initiation of the evaluation to determine the extent to which project-related performance information was available, as contained in the “Key Indicators of Success Report,” which is prepared and submitted by each recipient at the end of their project. The report template consists of a large range of indicators categorized under various themes such as prevention, monitoring, and community capacity building, among others. Performance data from all 71 completed projects, which program staff had recently consolidated, were used to assess the achievement of program outcomes.

This data collection method addressed evaluation questions 4, 5, 9 and 10.

Key Informant Interviews - Round I

A total of nineteen interviews were conducted with Environment Canada program management and staff, other government departments (OGD), and provincial and non-governmental organization stakeholders which were successful project proponents (a list of interviewees can be found in Annex D). A set of interview guides were prepared, which were provided to interviewees beforehand. These guides explained the purpose and provided the questions that were to be asked during the interview. For each interview, a set of interview notes was prepared. Interviews were conducted either in person or by telephone, and in the interviewee’s language of choice.

This data collection method addressed evaluation questions 1 through 11.

Key Informant Interviews - Round II

A total of 12 additional key informant interviews were conducted in July and August 2009, with program staff and Environment Canada departmental senior management (4), Interdepartmental Directors General IAS Steering Committee members (4), and federal department and agency partners (4). A list of interviewees can be found in Annex D. Interviews focused on the following issues:

This data collection method addressed evaluation questions 1, 2, 4 and 6.

Survey

An online survey of funding applicants (both successful and unsuccessful) was conducted between February 18, 2009 and March 9, 2009. The survey served as a validation tool for findings that were identified through interviews and document review. The survey was distributed to 100 proponents (72 successful, 28 unsuccessful) – a total of 37 responses were collected for a response rate of 34.7 per cent and 42.9 per cent, respectively (25 successful, 12 unsuccessful).

This data collection method addressed evaluation questions 2, 3, 6, 7, 9 and 11.

3.2.2 Limitations

There are five specific limitations associated with this evaluation.

Performance Information

Evaluators relied on consolidated performance information submitted by each recipient once their project is completed (71 out of 108 approved projects at the time of the evaluation) contained in the “Key Indicators of Success Report,” as well as on testimonies from key informants to assess the success of the program in meeting its immediate and intermediate outcomes. The information that was submitted by each recipient once their project is completed has not been validated by the program or by this evaluation; therefore, there is uncertainty about the accuracy of the performance data. Specifically, there are issues of duplication (i.e., double-counting) and inconsistency of performance data, as well as a disconnect between some performance indicators and the project-level outputs and outcomes that they are intended to measure. Thus, the evaluators’ ability to report on the achievement of program outcomes was limited.

Furthermore, performance information contained in the “Key Indicators of Success Report” regarding targets and percentage achieved of targets can be misleading. In many cases, targets were not established at the beginning of each project; therefore, it is difficult to accurately determine the level of success of a project in achieving its objectives.

Financial Information

The IASPP received a total of $5 million in funding over five years (approximately $1 million per year) for the contribution component of the program. No additional funding for program administration was approved (i.e., Operations & Maintenance and salary) and all operating expenses for the IASPP were covered by Environment Canada’s A-base budget. The financial information with respect to program administration was provided by the program lead. The breakdown of Operations & Maintenance costs is derived from the financial system but the allocation of salary costs to the program is a best estimate. Therefore, total operational expenses may not be completely accurate, which would affect the comparison of operational expenses to contributions used to examine program efficiency.

External Perspective

The input of individuals and groups who have not benefited from or been involved in delivering the program would have been useful to provide an external perspective on the program’s relevance, to explore possible limitations to the program’s reach, and to gain external views on the program’s perceived impacts to date. Given the scope of the evaluation (based on the low materiality of the program), such individuals were not engaged as part of the evaluation.

Scope of the Evaluation

Each of the participating departments and agencies undertook an evaluation of their respective IAS-related programs as per the national 2008 IASSC RMAF. The RMAF indicated that each department would conduct an evaluation of their respective programs. Thus, the scope of this evaluation was limited to examining EC’s participation in the IASSC through the IASPP. The evaluation did not examine any aspects related to the IASSC itself (e.g., shared governance, shared outcomes as per the 2008 IASSC RMAF).

Survey Sample Response

Although the survey response rate was 34.7 per cent for successful project proponents and 42.9 per cent for unsuccessful project proponents, actual numbers of respondents for each category are relatively low. Given the small number of respondents which limited the ability of the evaluation to generalize survey findings, the survey data should not be interpreted in isolation from other sources of evaluation data. The results of the survey, however, closely matched the findings in the key informant interviews and support the conclusions reached from other sources of evaluation evidence.

4.0 Findings

This section presents the evaluation findings by evaluation issue (relevance, success, cost-effectiveness, design and delivery) and by the related evaluation question. The findings at the overall issue-level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question based on a judgment of the evaluation findings. The rating symbols and their significance are outlined below in Table 1. A summary of ratings for the evaluation issues and questions is presented in Annex E.

Table 1: Rating Symbols and Significance
SymbolSignificance
AchievedThe intended outcomes or goals have been achieved or met
Progress Made; Attention NeededConsiderable progress has been made to meet the intended outcomes or goals, but attention is still needed
Little Progress; Priority for AttentionLittle progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis
N/AItems where a rating is not applicable
~Outcomes achievement ratings are based solely on subjective evidence


4.1 Relevance

Evaluation Issue: Relevance

Overall Findings: Interviewees felt that there is a continued need for a contribution program in Canada to address IAS at the grassroots level. However, although many felt that the individual projects funded by the program were successful and effective, they questioned the effectiveness of the program to achieve its overall objectives given issues related to a lack of concise and focused program priorities and limited program funding. Some interviewees felt that the funding could be diverted to other programs under the national strategy to provide a greater impact at addressing IAS-related concerns nationally.

Although the IASPP was well-aligned with government-wide priorities in 2005, when the program was created, as of the present, there is only an indirect link between IASPP objectives and current government-wide priorities (many interviewees spoke of the negative economic impact of IAS on various industries including fisheries, agriculture and forestry).

Overall, the IASPP does contribute to the national strategy. It complements other programs under the national strategy, in that it engages stakeholders beyond the federal and provincial governments, targeting community-level and grassroots organizations. The program is unique in that it is the only contribution program which provides funding to these organizations to take action related specifically to IAS.

Interviewees felt that if the IASPP did not exist, there would be also a decrease in coordination of IAS-related activities across the federal government. Environment Canada is perceived as the best situated department to administer the IASPP and coordinate the national strategy, given that it is seen as a neutral party without a vested interest with respect to IAS.

Evaluation Issue: Relevance

Q1: Does the IASPP continue to be consistent with government-wide priorities and the departmental mandate?

Indicator(s)
Methods
Rating
  • Program objectives that are aligned to government-wide priorities and departmental mandate.
  • Inter/intra-governmental relations and coordination (F/P/T and First Nations).
  • Document review
  • Interviews
Progress made, attention needed

Alignment to Government-wide Priorities

A review of relevant documents, including the Speech from the Throne(2005) and Budget 2005, reveals that the IASPP was consistent with government-wide priorities in 2005 when the program was created as part of the national strategy. Budget 2005 provided $85 million in funding for the national strategy for 5 years ($5 million was allocated for the IASPP over the same period). However, more recent documents do not provide any link between the IASPP and current government-wide priorities.

The majority of Round II interviewees felt that the objectives of the IASPP aligned to some extent with current government economic priorities. Specifically, many mentioned the negative economic impact of IAS on various Canadian industries, most importantly fisheries, agriculture, and forestry with respect to natural resources (e.g., lumber, crops), trade, as well as operating costs. In a 2003 study, examining 16 species, it was estimated that cumulative annual costs related to IAS within various sectors was between 13.3 and 34.5 billion dollars.5 A few individuals noted that although IAS is not “making the headlines” as it did when the national strategy was developed in 2004, the issue has not disappeared and continues to be a concern. One individual specifically discussed IAS in the context of the Species at Risk Act (SARA), stating that SARA is an important issue for government given the significant amount of funding that the federal government continues to provide to address species at risk issues. In 2002, it was estimated that about 24 percent of species at risk in Canada listed by the Committee on the Status of Endangered Wildlife in Canada may be threatened with extinction by IAS.6 Given that IAS is a significant threat to species at risk, it therefore continues to be an indirect government priority.

Contribution to IASSC

Round II interviews indicated the IASPP does contribute to the joint national IASSC. The national strategy aims to target investment of new funding to support science-based regulatory initiatives (including risk assessment and import controls), strengthen national surveillance for early detection, and raise public awareness and understanding of harmful practices that introduce IAS into Canada. The IASPP was included in the national strategy to complement the activities of other federal department and agency partners which primarily focused on science-based regulatory initiatives and national surveillance, while the IASPP was intended to promote public compliance with regulatory approaches and to establish voluntary initiatives to address pathways not amenable to regulation. Specifically, the activities funded through the IASPP are directly linked to the education and outreach component of the IASSC, specifically through the “Plan for Engaging Canadians and Sectors” output in the IASSC Logic Model.

Most interviewees indicated that the IASPP does complement the other activities of the national strategy. The IASPP specifically targets communities and organizations in order to develop IAS-related activities at the local level, thereby engaging stakeholders beyond the federal and provincial government, which fills a gap outside the scope of the activities of other federal departments and agencies. As one individual noted, “it finds a niche that has not been covered by the jurisdictions (federal or provincial),” focusing on raising awareness at the community-level and engaging grassroots organizations to take action related to IAS within their communities.

Alignment to Departmental Priorities

In terms of current departmental priorities, Environment Canada program staff and senior management discussed alignment to SARA (discussed above), the Migratory Birds Convention Act (i.e., impact IAS have on nesting areas of native birds, destroying habitat), Wild Animal and Plant Protection and Regulation of International and Interprovincial Trade Act (WAPPRIITA) and the United Nations Convention on Biodiversity.

WAPPRIITA is the legislative vehicle by which Canada meets its obligations under the Convention on International Trade in Endangered Species of Wild Fauna and Flora (CITES), by regulating the international and interprovincial trade in animals and plants. Environment Canada is the federal department responsible for the implementation of the Act, which provides the authority to designate species as invasive or harmful. While Environment Canada is responsible for terrestrial animals from a regulatory sense, Fisheries and Oceans Canada is responsible for regulating most aquatic species (in certain cases, Canadian Food Inspection Agency regulates aquatic plants7). The Fisheries Act does not provide authority to designate aquatic species as invasive or harmful (the only possible mechanism to do so is WAPPRIITA). However, as of present, no aquatic species has been designated under WAPPRIITA as invasive or harmful. Furthermore, one interviewee stated that the focus of WAPPRIITA is on endangered species not necessarily IAS.

The document review revealed that the IASPP is aligned with the requirements under the United Nations Convention on Biological Diversity, of which Canada is a signatory member. Canada joined the United Nations Convention on Biological Diversity in 1992 and developed the Canadian Biodiversity Strategy in 1995, with the goal of monitoring and controlling importation of alien species. Article 8 (h) of the Convention indicates that participating states must “prevent the introduction of, control or eradicate those alien species which threaten ecosystems, habitats or species.”

A review of Environment Canada’s corporate documents (Report on Plans and Priorities 2008-2009; Departmental Performance Report 2007-2008) provides evidence that the IASPP remains consistent with the department’s mandate. All interviewees also felt that the IASPP is most strongly aligned with the department’s priority related to the protection of biodiversity. IAS could lead to population declines in native species and reduced biodiversity. For instance, in the Great Lakes, sea lamprey have been implicated in the extinction of the Deepwater Cisco, and zebra mussels have extirpated native mussels from some areas.8 The objectives of the IASPP align with the departmental intermediate outcome “Biodiversity is conserved and protected,” which is linked to the departmental strategic outcome “Canada’s natural capital is restored, conserved and enhanced.” The approved departmental Program Activity Architecture for fiscal year 2010-2011 has also identified biodiversity as a departmental intermediate outcome, linked to the departmental strategic outcome “Canada's natural environment is conserved and restored for present and future generations.”

Interviews with other government departments also indicate the IASPP continues to be relevant. All representatives indicated that the IASPP aligns well with their own efforts and goals. Interviewees from Natural Resources Canada felt that the IASPP does align well with its own priorities. Specifically, the Canada Forest Service has a Forest IAS Program and a National Forest Pest Strategy that are based on the same priorities of the IASSC. The Canadian Food Inspection Agency sees the IASPP as complementary to its own activities and supportive of its departmental strategic outcome, “A safe and sustainable plant and animal resource base.” One CFIA interviewee stated that the “goals and objectives of the IASPP are very similar in that they want to reduce the spread of IAS in Canada.”

Evaluation Issue: Relevance

Q2: Is there a continued need for the IASPP?

Indicator(s)
Methods
Rating
  • Utility/rationale for the program
  • Ongoing need for capacity or activity in the area of departmental jurisdiction
  • Document review
  • Interviews
  • Survey
Progress made, attention needed

Continued Need for the IASPP

The prevention, early detection, rapid response and management of IAS are a continuing challenge in Canada. According to the International Union for Conservation of Nature9, an international organization dedicated to natural resource conservation, IAS are the second most significant threat to biodiversity after habitat loss. They often dominate the ecosystems they invade, causing losses to productivity and market access and increasing costs to producers.

Overall, Round II interviewees felt that there is an ongoing need for a contribution program in Canada to address IAS at the grassroots-level. All interviewees felt that IAS is an ongoing relevant issue and that there is a need for a contribution program aimed at engaging grassroots organizations and the public to undertake activities related to raising awareness of IAS. However, most interviewees stated that there are two significant related issues that the IASPP needs to address going forward: re-examining the priorities of the program, and program funding.

A majority of Round II interviewees stated that, although the goals of the program were relevant and the projects funded by the program were making a difference in their communities, they questioned the overall impact the program was making with respect to raising awareness at the national level. As one interviewee stated, there is a “need for a more structured approach focusing on critical issues and not funding tiny projects scattered across the country.” Most interviewees felt the program was using a “shotgun approach”, thereby spreading itself too thinly (i.e., providing nominal funding to many small localized projects spread across Canada instead of focusing on critical issues of importance related to IAS at the national level), which had a negative impact with respect to the overall effectiveness of the program to address IAS-related concerns nationally. As another interviewee stated, “there is a need to focus on a small number of priorities and make a bigger impact, not on many projects with little likelihood of really making a difference nationally.” At most, interviewees felt that there were marginal gains from the individual projects funded by the program at the local level (i.e., individuals reached, IAS addressed), not major changes.

Many interviewees discussed the need to determine a few key priorities for the program, where it can really make a difference. These included among others a focus on priority pathways and IAS (e.g., Emerald Ash Borer, Asian Longhorn Beetle), linking priorities of the program to the priorities of federal department and agency partners (i.e., Canadian Food Inspection Agency, Fisheries and Oceans Canada, Natural Resources Canada), linking IAS priorities to certain species (e.g., species at risk), or focusing on only one of the four key strategic goals of the joint national strategy, namely prevention, early detection, rapid response and management.

Furthermore, although most felt that the approach used by the program (i.e., projects funded through contribution agreements) was an effective approach to raising public awareness and engaging grassroots organizations, they also felt the overall effectiveness of the program was hampered by the total amount of program funding. Most felt that, at one million dollars per year of available contribution funding, the program could not achieve its overall objectives. As one individual noted, “to say that the program would fully address all aspects of IAS [i.e., prevention, early detection, rapid response, and management] is trying to make too much of the program because it is so small.” A few interviewees questioned whether the money could be used elsewhere to have a greater impact with respect to IAS in Canada. One individual stated that “a million dollars is a drop in the bucket. At a million dollars, cutting this program will have no effect on the national strategy.” Another individual noted that “for one million dollars, you’re not even in the game. You might as well spend it elsewhere as you can’t make enough of a difference.” Similarly, another individual stated that with such limited resources available, the funding could be used elsewhere to provide a greater impact on addressing IAS-related concerns. Although the relevance of the program was not questioned in and of itself, the ability of the IASPP to achieve its overall objectives given present funding levels was raised as a concern.

Gaps/Complementarity/Duplication of IASPP

Round II interviewees also discussed the gaps that would emerge if the IASPP did not exist. Most discussed the important contribution of the IASPP to the national strategy in terms of engaging local and community-level organizations. If the IASPP did not exist, interviewees felt that an important capacity to detect IAS at the community-level would be lost (e.g., monitoring program in Manitoba focusing on detecting Dutch Elm Disease, Brown Spruce Longhorn Beetle in Halifax, Emerald Ash Borer in Windsor). Without these grassroots organizations, it was felt that the strategy would lose an important component to address IAS. It was also felt that it would create a perception that government was not engaged at the community level.

Furthermore, interviewees felt that there would be a decrease in Environment Canada’s ability to fulfil its coordinating role with respect to IAS-related activities across the federal government. Most interviewees mentioned that Environment Canada was the best placed of all federal departments and agencies to take on the coordination role with respect to the national strategy. Most perceived the department to be an “honest broker with respect to national coordination and delivery.” Whereas the other departments and agencies were seen to have a vested interest with respect to IAS, either species-related (Fisheries and Oceans – aquatic; Canadian Food Inspection Agency – plant and plant pests) or sector-related (Agriculture and Agri-Food Canada – agriculture; Natural Resources Canada – forestry), Environment Canada, on the other hand, was perceived to have the broadest mandate to address IAS-related issues and to be a neutral party. As a result, interviewees felt that there was no other federal government department or agency that could take on the coordinating role with respect to the national strategy. Most interviewees also felt that Environment Canada should continue to administer the IASPP. By not having a vested interest, Environment Canada was considered to be the best situated to ensure that there existed one global vision and message related to IAS issues, whereas if it was parceled off, each department and agency would focus on their own specific interests.

Most Round II interviewees indicated that the IASPP is the only funding mechanism which is national in scope and focuses on engaging grassroots-level organizations to address issues related to IAS, through both action-oriented (e.g., pulling weeds) and information-oriented (e.g., developing and distributing pamphlets and brochures) projects. Stakeholder interviews also highlighted that there are limited funding opportunities to address IAS challenges at local/regional levels and that the IASPP was often the only funding source for IAS-related activities. This has created a high demand for the IASPP as evidenced by the number of applications received during each request for proposals, which average over 150 per year (to date, a total of 108 projects have been approved for funding, an average of 27 per year).

Table 2 provides some details with respect to the demand on the IASPP. The request-for-proposals for 2005-2006 and 2006-2007 were combined into one process due to delays in starting up the program and getting funds disbursed. On average, there are almost $8 million worth of proposals submitted each year for a program that provides, on average, $1 million per year in contributions. The amounts approved do not always reflect the planned distribution of $1 million per year. As mentioned, delays in the first year caused a combined call, and some of those projects received multi-year funding which decreased the amount available for 2007-2008.

Table 2: Result of Call for Proposals
 Combined Call
2005/06, 2006/07

2007/08

2008/09
Number of Applications310123192
Number of Approvals581832
Percentage Approval18.7%14.6%16.6%
Total Request (approx.)$19m$7m$7.3m
Total Approved Funding*$2,368,139$466,000$844,320

* Values are for total approved funding – see Table 13 for disbursed amounts.

Furthermore, without IASPP funding, many IAS-related projects are not implemented. Survey results revealed that of those project proponents who were not funded by the IASPP, 36% (4 respondents) stated the project was cancelled, 27% (3 respondents) stated the project was postponed, and 36% (4 respondents) said the project continued but at a reduced level.

Overall, interviewees felt that these gaps could not be filled by any other entity. Specifically, interviewees mentioned that provinces do not have the available resources to fund local projects. Furthermore, most felt that parceling off the program to specific federal departments/agencies would increase the likelihood of duplication of effort (e.g., invasive alien plants could be considered either the responsibility of the Canadian Food Inspection Agency or Fisheries and Oceans Canada) and could lead to gaps in the national strategy (one individual stated that Environment Canada “kept the federal family together” by bridging the gap between federal departments and agencies who would otherwise continue to work in silos).

When asked whether the program duplicates other programs in Canada, most interviewees stated that the program is unique. Although other programs may exist at the federal or provincial level which fund IAS-related activities (e.g., the Ontario Stewardship Fund, which does occasionally fund IAS-related projects, Agriculture and Agri-Food Canada’s Growing Forward10), the IASPP is the only contribution program which specifically addresses the issue of IAS at the national level. Most interviewees felt that the program is complementary to other programs in Canada, including Habitat Stewardship Program (HSP), which focuses on species at risk, and EcoAction, both administered by Environment Canada. These three programs coordinate their processes to ensure, for the most part, that IAS-related projects are not funded by either HSP or EcoAction (though the HSP has funded IAS projects related to species at risk). EcoAction, although a community-based contribution program also has limited funds to address IAS-related projects. However, a few interviewees felt that there is a clear overlap between the IASPP and the HSP. One individual indicated that the scope of the HSP could be marginally expanded to include the objectives of the IASPP. Specifically, the most important concern surrounding IAS in Canada is the negative impact they have on species which are listed as at risk, and therefore IAS-related projects should focus on those IAS which negatively affect species at risk.

4.2 Success

Evaluation Issue: Success

Overall Findings: Overall, interviewees noted the establishment of the regional/provincial councils as the program’s key accomplishment to date. These councils are important in that they facilitate partnerships between various key stakeholders and provide an important contribution to the development of regional priorities related to IAS.

The overall consensus of interviewees is that the IASPP is contributing to some extent to resolving the challenges related to IAS in Canada. However, given the present funding level of the program, most felt that these contributions resulted in marginal gains at the local/community level (with respect to informing and engaging Canadians, as well as addressing IAS), not major impacts at the national level.

In addition, although the program has been successful to some extent (at the local/community level) in raising the awareness of Canadians through training and outreach, as well as through the generation and dissemination of products related to IAS, these do not necessarily indicate that Canadians have become better informed or more engaged in activities on priority IAS issues as a result of the program. Without baseline data, it is difficult to determine the program’s impact in getting Canadians better informed and engaged in IAS-related activities. There is also less evidence to suggest that the program has been successful in achieving the immediate outcome “threat to natural capital reduced.”

Furthermore, given issues related to the accuracy of available performance data, it is also difficult to determine the level of success the program has achieved.

Evaluation Issue: Success

Q4: Are immediate and intermediate outcomes being achieved as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • Document review
  • File review
  • Interviews
~ Progress made, attention needed

Overall, Round II interviews indicated the establishment of the regional/provincial councils11 as the program’s key accomplishment to date (including Alberta, Ontario, Manitoba and Saskatchewan). Before the inception of the program, only BC had established an IAS-related council. These councils have facilitated the establishment of partnerships between federal and provincial governments, industry, non-governmental organizations and aboriginal groups. These councils have provided an important contribution to the development of regional priorities related to IAS.

Other key accomplishments mentioned by interviewees included raising awareness with targeted communities on priority pathways (e.g., recreational boaters and anglers regarding cleaning the hulls of boats and transferring baitfish between bodies of water, horticulturalists, and aquariums). However, it was mentioned that these accomplishments have not occurred in a comprehensive fashion nationally.

Another key accomplishment cited by interviewees was the creation of a network across the country and sharing of products between various key stakeholder groups (e.g., creation of a particular pamphlet related to a specific IAS).

Given the funding level of the program, however, most interviewees agreed that the program did not have the resources to achieve its objectives. Although there was success at the individual project level, overall, the program at most had marginal gains at the local level, not major accomplishments at the national level.

There are four immediate outcomes and four intermediate outcomes identified in the 2008 IASPP logic model. The following sections will address each one separately. Performance information is derived from the consolidation of the “Key Indicators of Success Report,” which is part of the final report for each project. In total, 71 projects out of the total of 108 projects that have been funded are complete and have submitted this report. As already noted, the information in these reports is not verified or audited so the accuracy of available performance data is uncertain. It can, however, provide some indication of the results of the program as a whole.

Evidence exists that projects are generating outputs which are contributing to immediate and intermediate outcomes. A project file review and an analysis of available performance information clearly show significant production of outputs. These are presented in the following sections under the respective outcome to which they relate.

Almost all interviewees indicated that the program is achieving the intended results to some extent. A few interviewees noted that they are not able to assess the results of the program given that performance information is not distributed. For example, members of the Technical Review Committee do not have any feedback as to the results of the projects that they reviewed and approved. They mentioned this would be helpful when reviewing the next round of applications in order to help them determine whether an organization was successful in the past in achieving its objectives and thus a more reliable proponent for future projects.

In terms of the survey results (Table 3), almost 80% of successful project proponents felt the IASPP was contributing to the reduction of IAS in Canada through its activities to at least a moderate extent. Unsuccessful proponents who were able to respond to the question also felt the program was contributing to at least a moderate extent. No respondents felt that the IASPP was having no impact.

Table 3 – Survey Results for Overall Success of IASPP
“To what extent has the IASPP contributed to the reduction of invasive alien species in Canada through its activities?”SuccessfulUnsuccessful
n=25n=12
1. To no extent00
2. To a moderate extent114
3. To a tremendous extent80
4. Don’t know57

Immediate Outcome 1: Threat to natural capital reduced

Evaluation Issue: Success

Q4.a: Is the threat to natural capital reduced as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
  • Survey
~ Little Progress, Priority for Attention

Interviewees were generally of the opinion that reducing the threat to natural capital was a less important outcome of the program, explaining that the main objective of the program was to increase Canadians’ awareness with respect to IAS. It was felt that raising awareness would lead to a reduction of the threat to natural capital. Project proponents interviewed felt that most of their project activities were focused on increasing awareness, which would then lead to action on IAS. Project proponents themselves were unclear as to whether this outcome was being achieved. Furthermore, in many cases in the interviews across all categories, it was commented that “threat to natural capital reduced” was very hard to measure as there was no performance information available on which to make an assessment.

The following project outputs (Table 4), however, demonstrate that action at the local level has contributed to some degree to reduce the threat of IAS. In total, the projects supported by the IASPP have undertaken activities related to a range of invasive alien species (215) and pathways (31).

Table 4: IAS targeted by projects
PlantWeedPlant PestAquatic InvertebrateAquatic VertebrateAmphibianOthers
647211184217

Table 5 presents performance data related to environmental indicators. There is a large amount of activity that is occurring across a wide range of IAS. Although the data do show projects are carrying out activities related to this outcome, given that the data have not been validated, it is difficult to determine the level of success in achieving this outcome. Furthermore, without baseline data providing a clear indication of the present threat (related to each project-level outcome used to measure the level of success in achieving this program outcome), it is difficult to determine the degree to which this is an issue and the extent to which the program has been successful in addressing this issue. For example, available performance data may state that 100 square hectares of forest has had an invasive plant removed, but give no indication as to how many hectares of land are suffering from this invasive plant.

Table 5: Environmental Indicators
MeasuresIndicatorsTargetAchieved%
Environmental management manuals or plans implemented / signedHectares7,06015,050213%
Number of plans1115136%
General Habitat improvementNumber of participants1,0071,320131%
Number of hectares improved10,75730,163280%
Vegetation and exotics removalNumber of participants8591,205140%
Number of hectares improved186693373%
Native plants, trees and shrubs plantedNumber of plants / trees / shrubs100,175350,494350%
Area M21,0452,799268%
Wildlife habitat created or restoredHectares576241095%
Species at risk protected or restoredNumber of species216800%
Other native species protected or restoredNumber of species2127129%
Non-native species removedNumber of species27254941%
Wetlands, land ecosystems created, restored or rehabilitatedHectares442452102%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Immediate Outcome 2: Improved capacity to detect and respond to IAS

Evaluation Issue: Success

Q4.b: Is there improved capacity to detect and respond to IAS as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
  • Survey
~ Progress made, attention needed

This immediate outcome is very similar to the intermediate outcome of “improved community capacity, skills and knowledge with respect to IAS initiatives”. Both are equally hard to measure given there is no baseline of existing capacity which could provide a comparison against which to measure improvements. It is therefore difficult to assess the achievement of this outcome.

EC program staff could not speak to the issue of increased capacity at the local level (although it was mentioned that smaller communities have been made more aware of issues related to detecting and responding to IAS). Comments from other government departments were also very limited due to a lack of knowledge with respect to performance information related to the projects. One interviewee stated that there was most likely an improved capacity to detect IAS as a result of the program, although not necessarily to respond to IAS (as funding was targeted more towards improving capacity to detect IAS). In another case (related to unwanted plants targeted by the horticulture industry), the respondent was very positive and stated that the increased information with respect to IAS had indeed resulted in improved capacity to detect IAS.

Despite a lack of baseline data, the projects have reported on outputs in this regard, as demonstrated in Table 6 (Training). Performance data indicate that 3,617 individuals have received training. It is important to note that it is possible that duplication in these numbers has occurred (i.e., the same individual could have received training several times). The number of people who have received training more than once is likely low however given that the data are derived from 71 projects spread across the country.

Although projects have provided training opportunities for community members, and the projects have been successful in terms of the number of training sessions held and the number of individuals reached, an increase in training and the information provided through training does not necessarily indicate an improved capacity to detect and respond to IAS. These figures only indicate the potential for improved capacity.

Other performance information used to measure the level of success with respect to improved community capacity (Table 7) included how the projects were able to establish means and mechanisms to combat IAS and how effectively they were able to engage people in the process. However, there is a disconnect between the project-level outputs and outcomes identified for this immediate outcome and the performance indicators identified to assess their level of achievement. Although the program may have been successful in reaching and engaging individuals, these performance indicators will not allow the program to measure the level of achievement with respect to establishing a coordinated public monitoring network, establishing capacity of diagnostics and taxonomic expertise, or monitoring results and impacts of introductions. Therefore, it is difficult to determine whether the program has indeed been successful in improving the capacity to detect and respond to IAS.

Table 6: Training on IAS
MeasuresIndicatorsTargetAchieved%
Training opportunities for community membersNumber of training sessions held63140222%
Number of days of courses2789328%
Number of people1,5603,617232%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Table 7: Community Capacity
MeasuresIndicatorsTargetAchieved%
Established a coordinated public monitoring network to detect and report IASNumber of people engaged11,46520,875182%
Number of people reached48,099105,011218%
Established capacity of diagnostics and taxonomic expertise to identify IAS.Number of people engaged1975,7962942%
Number of people reached67626,5933934%
Monitored results and impacts of introductionsNumber of people engaged225270120%
Number of people reached1,4201,822128%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Immediate Outcome 3: Canadians better informed about IAS

Evaluation Issue: Success

Q4c: Are Canadians better informed about IAS as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
  • Survey
~ Progress Made, Attention Needed

All interviewees stated that the program was very successful at informing Canadians about IAS. EC management and staff, as well as other government departments all agreed that the IASPP had increased awareness related to IAS. Some EC staff mentioned that the support to IAS Plant Councils has been a major achievement and that, generally, one of their principal roles is to increase awareness of IAS in their respective regions. Interviewees across all categories mentioned the quantity of information products that have been produced and disseminated. Proponents were also confident that the program had achieved this outcome.

Performance data indicate that of the 71 projects that have been completed, 60 of them (or 84.5%) produced and disseminated information products related to IAS. Almost all (92%) of successful project proponents who responded to the survey indicated that the main objective of their project was to “Improve Canadians’ understanding and awareness of IAS and the individual actions and choices that contribute to their spread.”

Table 8 presents the performance information on information products produced, although it is not clear from these figures the actual number of products generated versus distributed.

Table 8: Information Products
MeasuresIndicatorsTargetAchieved%
Public information generated/disseminatedNumber of products generated/ disseminated26,41060,072227%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

While production of information products is one area of considerable strength of the IASPP, outreach figures (Table 9) should be treated with some caution. There was very limited orientation given to proponents on the definition of the performance indicators (e.g., defining people reached versus people engaged). Thus, the reliability of the data being provided by individual proponents at the end of their respective projects is brought into question. Furthermore, the project-level outputs and outcomes, and their related performance indicators, point towards an increase in outreach and information generated and disseminated, not necessarily to Canadians being better informed. Although the program has been successful in increasing available information related to IAS through its production and outreach activities, it does not necessarily mean that Canadians are better informed, only that more information is publicly available.

Table 9: Outreach Indicators
MeasuresIndicatorsTargetAchieved%
Worked with mediaNumber of media interviews47119253%
Number of reports on projects2757211%
General public outreachNumber of people reached307,749290,81194%
Number of people engaged10,41011,524111%
Perform targeted outreachNumber of people reached138,817194,696140%
Number of people engaged21,38734,152160%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Immediate Outcome 4: Partners achieve measurable results

Evaluation Issue: Success

Q4d: Partners12 achieve measurable results as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
~ Achieved

Out of the 108 projects that have been approved since the program began in FY 2005-2006, 71 have been completed. The “Key Indicators of Success Reports” from these completed projects provide evidence that project proponents are achieving measurable results13, despite concerns surrounding the accuracy of performance data.

From interviews with EC senior management and staff, the overwhelming response was that the key result of the program has been an increased awareness of IAS in Canada. Comments such as “awareness is the really big contribution” were common among these interviewees; many mentioned the quantity of information products produced by the projects. Stakeholders interviewed commented that they have seen many good results and provided the establishment of IAS Plant Councils as an example and how that has helped bring all the IAS-related groups together.

A few of the interviewees from all three categories commented that the size of the fund may limit the results that can be achieved. The projects are relatively small in size and disbursed widely across the country. Issues around the timely transfer of funds were also raised as having the potential to limit results (i.e., some projects have a small window of opportunity related to their geographic location or the time of year they need to be implemented).

Intermediate Outcome 1: Improved jurisdictional coordination and communication of community response to IAS

Evaluation Issue: Success

Q4e: Is there improved jurisdictional coordination14 and communication as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
~ Achieved

The IASPP has contributed to improved jurisdictional coordination and communication. The program has provided funding to provincial government organizations and three provincial IAS Plant Councils--namely British Columbia, Alberta and Manitoba. Another funded provincial organization is the Native Plant Society of Saskatchewan. Smaller regional councils have also been supported, for example, the Central Kootenay Invasive Plant Committee in British Columbia. Two projects are also presently being supported whose objectives are to establish IAS Councils--one in Nova Scotia and one in the Yukon. The provincial and regional IAS Councils themselves play a large role in jurisdictional coordination and communication by networking IAS-related stakeholders at the provincial, regional and local levels, and sharing information and research.

Interviews with EC program staff, other government departments and proponents all mentioned the important contribution of the program in establishing these councils. In some cases, the IASPP has helped draw provincial attention to the issues of IAS; it has also allowed provinces to become more engaged in this area. No study of other IAS funding sources at the provincial level was undertaken, but comments from one provincial partner mentioned that the IASPP had allowed them, through their funded projects, to start activities related to IAS. One survey respondent mentioned the following as a key strength of the program: “the program is one of the only funding avenues for groups doing invasive alien species work. The IASPP has enabled many Canadian provinces to establish invasive species councils, which will empower them to better help the federal government meet the priorities of the IASSC.”

Other government department interviewees also raised the point that the IASPP, through the review of proposals process, has provided their staff at headquarters offices the opportunity to improve their understanding and knowledge of priority IAS issues at the local and regional levels.

While it is beyond the scope of the IASPP program, it is worth mentioning that some discussions have occurred about establishing a national IAS council. If and when such an organization is formed, it would be an appropriate mechanism to share the products and outputs that have come from the IASPP so that information can be disseminated nationally instead of just provincially or regionally.

Intermediate Outcome 2: Development and application of innovative techniques, practices, and/or processes involving introductions through unintentional pathways

Evaluation Issue: Success

Q4f: Is there development and application of innovative techniques, practices and/or processes involving introductions through unintentional pathways as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
~ Progress Made, Attention Needed

Some Environment Canada staff commented that science was not a major focus of the program during this initial five-year funding period, and that, in the future, it might be worthy to examine the relationship between the IASPP and the academic/scientific sector to increase activities in the area of developing new and innovative methods to address IAS. Program staff commented that they are aware of tools that have been developed based on their own knowledge of the projects. One specific example provided was an IAS database developed in Ontario.

Table 10 presents the performance indicators related to the development of new techniques and approaches. Information was collected with respect to the development of new tools to better predict IAS, new methods/technologies for preventing the spread of IAS and on new diagnostic tools to identify IAS. It is possible that what has been reported here is “new” in the sense that it has been developed and applied in a region for the first time, but not necessarily new in the sense that the specific method or tool may have already been used in some other region. An example would be the use of GPS systems to map out areas of IAS infestation. It is difficult to determine the actual quality of the outputs reported below. There also seems to be some inconsistency in the performance data. For instance, under the measure “Conducted and supported research aimed at developing tools to better predict IAS,” the number of reports distributed (1,217) is greater than the number of people reached (1,000), which may again be a result of duplication (i.e., double-counting), although it was not possible for the evaluation to verify this. Another example of an inconsistency from the performance data on innovation relates to the measure “Develop new diagnostic tools to identify IAS.” There were 36 new tools developed (forty-three percent of the target), while the number of new tools put into action is 483.

Table 10: Indicators on innovation
MeasuresIndicatorsTargetAchieved%
Conducted and supported research aimed at developing tools to better predict IASNumber of stakeholders involved4958118%
Number of tools developed2128133%
Number of reports distributed4051,217300%
Number of people reached01,000100%
Conducted and supported research on new methods/technologies for preventing spread of IASNumber of stakeholders involved221237107%
Number of people reached8,2458,308101%
Number of new methods or technologies developed or used1213108%
Develop new diagnostic tools to identify IASNumber of tools developed833643%
Number of people engaged298364124%
Number of people reached1,16647240%
Number of new tools put into action231483209%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Intermediate Outcome 3: Canadians are engaged in activities on priority IAS issues

Evaluation Issue: Success

Q4g: Are Canadians engaged in activities on priority IAS issues as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
  • Survey
~ Progress Made, Attention Needed

Interviews with program staff did not provide any concrete examples, but all staff felt that there was increased engagement of Canadians as a result of the communication and outreach activities of the projects. One proponent mentioned that since the IASPP has been operational, there has been a “groundswell of interest” on IAS issues. Another proponent commented that since his involvement from the inception of the program, there has been a progression from “almost no awareness to many groups now getting involved.”

Table 11 presents performance data related to engaging Canadians. While the figures below have not been validated, survey information indicated that almost all of the organizations that received IASPP funding rely on volunteers – only 12% of respondents indicated that they seldom use volunteers, and almost 30% of respondents stated they use over 50 volunteers on their projects. Performance data indicate that 3,831 volunteers participated in the projects funded by the program, an average of 35 per project. However, these figures are not identical to the figures related to volunteer participation provided in Table 12 related to community capacity. Although the number of volunteers is similar, the number of person-years is significantly different.

IASPP projects have also involved federal departments, as well as provincial and territorial departments. However, there seems to be duplication (i.e., double-counting) with respect to available performance data presented in Table 11. For instance, a total of 88 federal departments (62 being the target) and 116 provincial and territorial departments were reported to be involved in the projects. Similarly, nothing precludes more than one project proponent reporting that their project affected the same individuals, indicating that some of the people affected by a project may have been counted twice, which may explain why that particular project-level outcome achieved 684% of its target.

It is clear that one of the key strengths of the program has been raising Canadians’ awareness of IAS issues. Although the program has been successful in raising the awareness of Canadians through training and outreach, as well as through the generation and dissemination of products related to IAS, and it has made progress in achieving the immediate outcome, “Canadians better informed about IAS,” these do not necessarily indicate that Canadians have become engaged in activities on priority IAS issues as a result of the program. These only speak to the potential for Canadians to become more engaged because they are more aware and better informed. Furthermore, without baseline data, it is difficult to determine the program’s impact in getting Canadians engaged in IAS-related activities.

Table 11: Indicators on engaging Canadians
MeasuresIndicatorsTargetAchieved%
Volunteers participating directly in projectNumber of volunteers2,0053,831191%
Number of person-years1,01051851%
Federal departments involved in projectNumber of departments6288142%
Provincial or Territorial departments involved in projectNumber of departments93116125%
Municipal governments involved in projectNumber of governments88191217%
People affected by projectNumber of people161,2681,102,793684%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Intermediate Outcome 4: Improved community capacity, skills and knowledge with respect to IAS initiatives

Evaluation Issue: Success

Q4h: Is there improved community capacity, skills and knowledge with respect to IAS as a result of the program?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy
  • File review
  • Interviews
  • Survey
~ Progress made, attention needed

Interviews with program staff indicated that they believe community capacity has increased mostly as a result of the communication and outreach activities that have occurred.

The survey collected information on the organizational profile of the proponents. Approximately 42% have annual budgets of less than $1 million, whereas 36% have an annual budget over $1 million (22% were not able to respond). Approximately 56% have fewer than 20 staff members, whereas 38% have over 20 staff (about 6% were not able to respond). Given that many IASPP recipients are thus of a relatively smaller size, these organizations are more dependent on IASPP funding to carry out activities; therefore, IASPP funding can have an important impact on their capacity to undertake projects related to IAS.

Performance data presented in Table 12 below indicate that project proponents are engaged in their communities through public events, responding to inquiries through the provision of services, getting volunteers engaged and training them. However, these figures only indicate the potential for improved capacity. As was the case with the intermediate outcome related to Canadians becoming engaged in IAS-related activities, an increase in the amount of training, outreach, as well as the generation and dissemination of products related to IAS indicates a potential for improved capacity, skills and knowledge with respect to IAS. These performance indicators do not speak necessarily to improved capacity being achieved. Although the program has been successful in producing these outputs, the impact they have had on community capacity, skills and knowledge is uncertain.

Table 12: Indicators on community capacity
MeasuresIndicatorsTargetAchieved%
Public attendance at project eventsNumber of events330468142%
Number of people in attendance128,450146,392114%
InquiriesNumber of inquiries about projects4,98510,479210%
Services provided in response to requestsNumber of services provided467744159%
Number of people served2,3534,614196%
Volunteer participationNumber of volunteers2,2073,661166%
Number of volunteer person-years29108379%
Recognition of community participationNumber of awards66100%
Training opportunities for community membersNumber of training sessions held63140222%
Number of days of courses2789328%
Number of people1,5603,617232%

Note: Performance data contained in the table are as submitted by projects and have not been validated by the program or this evaluation.

Evaluation Issue: Success

Q5: Have there been any unexpected outcomes (positive or negative)? Were any actions taken as a result of these?

Indicator(s)
Methods
Rating
  • Documented unexpected outcomes
  • Lessons learned or best practices have been documented
  • Document review
  • File review
  • Interviews
N/A

No unintended outcomes were identified through this evaluation.

4.3 Cost-Effectiveness

Evaluation Issue: Cost-Effectiveness

Overall Findings: Evidence collected indicates that the program is being delivered in a cost-efficient manner through its low administrative costs (13.4%), as well as its significant leveraging of matching funds ($1.19:1 ratio).

Overall, interviewees indicated that the program’s approach was appropriate. No alternative approaches to a contribution program were identified that could be more cost-effective or efficient at achieving the same objectives as the IASPP. However, a few modifications to the approach presently implemented by the program were suggested, including the creation of two separate streams of IASPP funding, one for provincial governments and the other for non-governmental organizations and industry. A few interviewees also discussed the need to focus more on an action-oriented approach to addressing IAS in Canada.

It was felt by most interviewees that limited operational funding had a negative impact on the overall effectiveness of the program to achieve its objectives. Specifically, the lack of operational funding affected both the management of IASPP contribution agreements (i.e., timely transfer of funds, verifying individual project results), as well as Environment Canada’s coordinating role with respect to the national strategy.

Evaluation Issue: Cost-Effectiveness

Q6: Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?

Indicator(s)
Methods
Rating
  • Financial management system lined to results management process.
  • Consideration of alternate approaches
  • Document review
  • Interviews
Achieved

Program Efficiency

Cost-effectiveness is ideally measured by calculating the costs of achieving the measured program results and comparing them to the costs of achieving the same results through different means. Given the challenges encountered in identifying accurate performance data, effectiveness was measured indirectly by examining program efficiency and key stakeholders’ perceptions of the program’s efficiency and not a subjective assessment of cost-effectiveness.

Table 13 below presents a financial summary of the IASPP. It should be noted that financial information related to salaries is a best estimate provided by the program (given that no specific funding was earmarked for the administration of the program). In addition, the actual value for contributions reported here is different from Table 2. Table 2 presented approved values, while Table 13 presents disbursed values.

Table 13: Cost of the IASPP
Cost Items2006-072007-082008-09Total
A. Actual Operating Costs:
Salaries and Benefits (estimated)$97,804$120,107$97,160$315,071
Operating Costs$8,996$122,070$30,705$161,771
Sub-Total (A):$106,800$242,177$127,865$476,842
B. Non-Administrative:
Contributions15$624,894$1,958,592$984,185$3,567,671
Sub-Total (B):   $3,567,671
Total Costs (A+B):   $4,044,513
Ratio of total actual operating costs to contributions17.1%12.4%12.9%13.4%
OGD In-Kind Estimates (person/days)883554177

The IASPP was planned to be a $1 million per year contribution program. No additional funding for program administration was approved and all operating expenses for the IASPP have been covered by Environment Canada’s A-base budget. Given the level of resources currently applied to administer the IASPP, the program has proven to be cost-efficient at requesting, reviewing, and identifying projects that contribute to the overall goals of the program. It must also be noted that significant in-kind contributions are made by the TRC members (estimated at 177 person days thus far) related to the review and assessment of project proposals which occur on an annual basis.

It is difficult to match approved values with disbursed values and with the $1 million in funding for contributions that the IASPP was allocated each year over the five-year period. The delays in the first request-for-proposals in FY 2005-2006 resulted in a combined call that permitted multi-year funding. This, therefore, also had an impact on subsequent financial years.

Over the three years for which financial information is available, the percentage of total actual operating costs compared to total actual contributions is 13.4%. This ratio is similar to that of the Habitat Stewardship Program, whose administrative costs represent 13% of its contributions for the period 2004-2005 to 2007-2008 (budgeted amounts).

The workload of such a program is created by the volume of applications received and the subsequent administration of the approved projects. The request-for-proposal process is a time-consuming process, even with the implementation of an online application system. Program staff commented that they receive many queries from proponents during this period and do their utmost to respond to them all. There is also the screening and selection of an average of 150 proposals which are submitted per year. Other government departments contribute their staff time and resources to this process, as noted in the above table.

There is also significant leveraging of funds. Projects were selected based in part on their ability to leverage partnerships and funds. Information collected from the program indicates that for every dollar of IASPP funding, $1.19 is leveraged by the proponent.

With respect to the overall efficiency of the program, the overriding concern is with delays in transferring funds to project proponents. This was raised by almost all interviewees across all categories, as well as by survey respondents. Given the time-sensitive nature of IAS field activities, which usually have to be undertaken in the April to October time period, delays in funding can have a major impact on budgets and therefore project workplans. The need to revise workplans, budgets and contribution agreements due to delays in final approvals is perceived to be a major weakness in the program.

Effects of Lack of Operational Funding

Interviewees discussed various issues to emerge as a result of the program not having specific operational funding. Overall, many felt that a lack of operational funding had a negative impact on the overall effectiveness of the program to achieve its objectives. Specifically, the lack of operational funding affected both the management of IASPP contribution agreements, as well as Environment Canada’s ability to fulfil its coordinating role with respect to the national strategy. As one individual stated “without question, operational funding with dedicated personnel and in-house resources would certainly increase the effectiveness of the program.”

A few interviewees felt that this lack of operational funding was especially problematic for the program at the beginning. For instance, there were major delays in the approvals process for the first round. Those who were approved did not see funding until the second round was well underway. In some cases, given timing issues, projects could not undertake the activities that they proposed. Other issues noted by interviewees included:

  • getting contribution agreements out in a timely fashion;
  • an inability on the part of the program to follow-up with recipients to determine whether or not projects had really achieved the expected results they described in their “Key Indicators of Success Report”; and
  • delays in providing feedback to non-recipients due to capacity issues (as resources were used to prepare contribution agreements for recipients).

Furthermore, the lack of operational funding also detracted from the ability of the program to coordinate the national strategy. Environment Canada was tasked with the responsibility to provide overall leadership and coordination of the national strategy. Given a lack of resources and capacity, the program has spent much of its time administering the IASPP contribution agreements and, as a result, has not been able to direct resources to its coordination role. Participants16 who attended the National Invasive Alien Species Forum, held in June 2009, felt that there was a gap in national governance and that a national coordinating/governance mechanism needs to be implemented (this included the need to establish priorities, the need to clarify roles and responsibilities related to the national strategy and the need to create a mechanism to ensure partnerships among the key stakeholders). A few Round II interviewees also noted the lack of an interdepartmental coordination and governance structure for the national strategy as a result of the lack of operational funding.

Alternative Approaches

In terms of alternative approaches, most interviewees felt the program’s approach was appropriate. For the most part, no alternatives were identified that could be more cost-effective or efficient at achieving the same objectives as the IASPP. In some cases, interviewees suggested that modifications could be made in a subsequent iteration of the program. For example, one stakeholder suggested that it is not appropriate for provincial governments to be competing alongside small organizations (e.g., a First Nations community organization), and that provincial needs should be addressed in a different manner. For example, two streams of competitive funding could be created, one for provincial governments and the other for non-governmental organizations and industry.

A couple of interviewees discussed the need for a more action-oriented approach to addressing IAS in Canada. As one interviewee noted, “at some point you’ll saturate the market with pamphlets and brochures. It’s still better than nothing, though we do not have the impact we could have.” Another interviewee felt that there was a need for more action-oriented projects to deal with introductions that have already occurred in order to minimize the impact of those introductions. The IASPP was designed to focus on IAS-related projects which improved the understanding and awareness of Canadians of IAS, as well as enabling Canadians to become actively and concretely involved in projects that address the threat of IAS.

A few questioned the mix of funding between prevention, detection, response and management. Most EC management and staff and OGDs were comfortable with the emphasis on prevention and detection as this falls within the federal jurisdiction. A few of the stakeholders voiced their interest in seeing more work on management, although they were in the minority.

One final area for improvement mentioned was the distribution of the performance information to other government departments involved in the program, for example, the TRC members. This would allow them to know the performance of project proponents on past projects and assist in the project review and selection process.

4.4 Design and Delivery

Evaluation Issue: Design and Delivery

Overall Findings: The program is being delivered as designed and continues to make operational process improvements (e.g., creation of a website, addition of an online application system). The program’s activities, outputs and expected outcomes are clearly identified although they could be better communicated to all stakeholders. A governance structure exists that is consistent with the overall objectives of the program and the IASSC.

There appear to be areas for improvement regarding the design and delivery of the program. Specifically:

1) Accuracy of available performance data

There is uncertainty about the accuracy of available performance data for the IASPP, which is used to assess the level of success achieved for particular outcomes. Although performance data are being collected and reported by project proponents at the completion of their projects, the information has not been validated by the program. There are issues of duplication (i.e., double-counting) and inconsistency of available performance data, as well as a disconnect between some performance indicators and the project-level outputs and outcomes that they are intended to measure. Thus it is difficult to determine the level of success of the program in achieving its expected outcomes and overall objectives.

2) Full implementation of performance management plan

The performance management plan has not yet been fully implemented. Although the program does collect performance data, this information has not been analyzed to determine the extent of the program’s achievement of its expected outcomes and overall objectives. Therefore, performance data are not currently being used to inform program decision-making.

3) Timeliness of funding disbursement

Providing successful proponents with funding for approved projects in a timely manner was identified as an issue by all three categories of key informants as well as by a majority of survey respondents.

4) Communications to proponents with respect to the overall objectives of the IASPP

Interviews with project proponents revealed that they were less clear on the overall objectives of the IASPP. The application guidelines do provide some guidance, albeit at a very high level. They do not specifically state the objectives or the expected results of the IASPP.

Evaluation Issue: Design and Delivery

Q7: Are program deliverables and expected outcomes identified clearly, and is the program being delivered as designed?

Indicator(s)
Methods
Rating
  • Demonstration of the program’s expected deliverables and results
  • Consistency with documented design and delivery/implementation
  • Document review
  • Interviews
  • Survey
Progress made, attention needed

Q3: Are planned activities being implemented and producing expected outputs?

Indicator(s)
Methods
Rating
  • As per indicators in the performance measurement plan and reporting strategy

The program deliverables and expected outcomes are clearly identified in the RMAF and IASPP logic model. Interviewees from other government departments indicated that they have a clear understanding of the expected outcomes of the program. Program staff and other government departments commented that the program was generally being delivered as designed.

For instance, with respect to the contribution process, there have been four requests for proposals that have taken place since FY 2005-2006, and 625 projects have been reviewed, of which 108 have been approved and for which contribution agreements have been prepared and signed (up to FY 2008-2009). Although proponents do provide a “Key Indicators of Success Report” at the end of their project, performance data have only recently been consolidated. No analysis of the data has been conducted to date, and the performance data have not been used to inform program decision-making (this issue is discussed in further detail in Question 9).

In FY 2007-2008, the program implemented an online application system. EC program staff commented that this system was a vast improvement to the previous process where applications were submitted in hard copy to the program. One possible area for improvement noted by the evaluators during the document review was the lack of clear proposal evaluation criteria included in the application documents. Understanding the evaluation criteria and ranking system related to the assessment of project proposals would assist proponents in the development of proposals and improve the transparency of the selection process.

A new website has also been developed that provides information and support to Canadians on the IASSC and details on the IASPP, including a summary of previously funded projects, success stories and the program’s objectives and goals. Further improvements are planned and will be implemented pending the availability of time and resources. The improvements include introducing an electronic reporting system within the current online application system. This would allow project proponents to report online and assist in the capture and analysis of performance information. Discussions pertaining to automating processes for the drafting of contribution agreements, which should reduce the turn-around time needed to produce and/or revise agreements, have also taken place.

Interviews with proponents revealed that they were less clear on the overall objectives of the IASPP. The application guidelines do provide some guidance, albeit at a very high level, and they place an emphasis on the evaluation criteria and list examples of the types of activities that are funded. They do not specifically state the objectives or the expected results of the IASPP. However, all program staff and other government department representatives interviewed felt the objectives of the program were clear. This perception of clarity could be due to a greater familiarity with government IAS programs themselves, either through their own departmental programs or close interaction with the IASPP.

Survey questions focused specifically on the IASPP’s application, selection and funding processes. The results (Table 14) reveal that, in general, the application and selection processes are operating well, while there were issues surrounding the funding process.

Most successful proponents (91%) felt that, overall, the application process was clear, while 64% of unsuccessful proponents felt it to be clear. The survey revealed that both successful and unsuccessful proponents were clear on eligibility requirements17 (91% and 88%, respectively). Results were significantly lower when respondents were asked whether they received information, guidance and support provided from program staff during the application process. While 65% of successful proponents responded that they had received information, guidance and support, only 9% of unsuccessful proponents felt the same. This is at odds with interviews of program staff and other government department representatives, who stated that great effort is made to respond to all queries received during the request-for-proposal process.

Regarding the selection process, the majority of successful and unsuccessful proponents were clear on the selection criteria18 (78% and 64%, respectively). Feedback as to why projects were not funded seemed to be an issue. Only 27% of unsuccessful proponents felt they received an explanation from the program on why their project was not funded.

The timely transfer of funds was identified in almost all of the interviews from all three categories as an area for improvement (i.e., getting “money out the door”). This was also highlighted in the survey. Although 74% of successful proponents responded that the funding process was clearly articulated in the contribution agreement, only 35% of successful proponents stated that they received the funds in a timely fashion. This is a critical issue given the time-sensitive nature of IAS activities that generally occur during the field season, roughly April to October each year. The delays in getting funding to recipients has had an effect on the workload of proponents and program staff because of the need to revise workplans and budgets due to funding arriving late in the season in which field work for IAS can be conducted.

Table 14: Survey results related to application process
 Successful

n=25
Unsuccessful

n=12
YNDKYNDK
1. Based on your experience preparing an application for IASPP funding, was the overall application process clear?21

91%
207

64%
31
2. Did you receive information, guidance or support from IASPP representatives during the application process?15

65%
7119

82%
1
3. Was the information and guidance provided on the IASPP website sufficient to help your prepare your IASPP application?21

91%
206

55%
32
4. Based on your review of program eligibility criteria in available program documents, did you find these criteria to be easy to understand?21

91%
208

88%
12
5. Based on your review of program selection criteria in available program documents, did you find these criteria to be easy to understand?18

78%
327

64%
22
6. Was the decision regarding whether your funding proposal was successful or not communicated to you in a timely manner?18

78%
326

55%
41
7. Were the timelines related to your project’s key deliverables clearly articulated in the contribution agreement?19

83%
31N/A  
8. Was the funding process clearly articulated in the contribution agreement?17

74%
33N/A  
9. Were the funds provided to you in a timely manner?815

65%
0N/A  
10. Did you receive an explanation from EC on why this project was not funded through the IASPP?N/A  37

64%
1
11. Were the reasons your IASPP funding application was not successful made clear to you?N/A  28

73%
1

Note: Tallies may not add up to total respondents as some respondents skipped certain questions.

Evaluation Issue: Design and Delivery

Q8: Are activities and outputs linked to shared outcomes and the overall government-wide objectives?

Indicator(s)
Methods
Rating
  • Links between activities, outputs and outcomes supporting joint national objectives
  • Document review
  • Interviews
Progress made, attention needed

Although the logic model describes program activities and outcomes, the relationships between the activities of the program and the outcomes are unclear. Furthermore, key outputs are not included in the logic model although they are described in the RMAF itself. Therefore, the format of the logic model does not allow for the individual activities and outputs to be linked to the program’s immediate outcomes. It also makes it difficult to link program activities not only to the program’s expected outcomes but also to the overall government-wide objectives. As a result, the program’s underpinning logic is not expressed in the logic model as clearly as it should be.

Program staff and other government departments noted areas for improvement related to the expected outcomes as they are presented in the program’s logic model, specifically the placement of certain outcomes at either the immediate or intermediate level. For example, it could be argued that the main focus of the program has been to produce information as well as to engage and raise the awareness of Canadians related to IAS.

The IASPP was designed and is delivered to support the implementation of the IASSC with an emphasis on raising the awareness of Canadians and engaging them in IAS activities, both of which are identified outcomes of the program. In addition, project eligibility and selection is consistent with the IAS priorities of prevention, early detection, rapid response and management. A review of the projects indicates that all projects address at least one, and usually multiple, IAS priorities. Table 15 presents the alignment of projects to IAS priorities. The strongest emphasis has been on prevention and detection (86% and 91.5% of approved projects address those IAS priorities, respectively), while 20% and 50% of projects address rapid response and management, respectively. As stated in the IASSC, “as predominant landowners and managers of agriculture, forests, freshwater fisheries, and wildlife, the provinces and territories could be seen to have broad responsibilities for the management - eradication, containment, and control - of established and spreading invaders”. The emphasis on prevention and early detection is, therefore, consistent with the federal role.

Table 15: IAS Priorities
PriorityPreventionEarly DetectionResponseManagement
Number of projects61651435
Percentage86%91.5%20%50%

Total number of projects = 71

Evaluation Issue: Design and Delivery

Q9: Are performance data collected against program activities/outcomes and used for informed decision making?

Indicator(s)
Methods
Rating
  • Measuring and reporting strategies and procedures
  • Presence of populated performance data system
  • Decisions based on performance information
  • Document review
  • File review
  • Interviews
  • Survey
Little progress, priority for attention

Performance data are collected against program activities/outcomes as evidenced by quarterly expenditure reports, interim project progress reports, as well as the end-of-project reports which include the “Key Indicators of Success Report.” Based on an examination of consolidated performance data from these reports, the program presently collects information on approximately 56 project-level outputs and outcomes with 114 indicators.

There were issues raised in the “Success” section of the present evaluation with respect to the accuracy of available performance data used to assess the level of success achieved for particular outcomes. Specifically, there appear to be instances of duplication (i.e., double-counting) and inconsistency in performance data (e.g., under the intermediate outcome “Development and application of innovative techniques, practices, and/or processes involving introductions through unintentional pathways,“ the number of reports distributed is greater than the number of people reached). There is also a disconnect between the project-level outputs and outcomes and the performance indicators identified to measure their level of achievement (e.g., under the immediate outcome “improved capacity to detect and respond to IAS,” measuring the number of individuals reached and engaged does not allow the program to measure the level of achievement with respect to establishing a coordinated public monitoring network, establishing capacity of diagnostics and taxonomic expertise, or monitoring results and impacts of introductions). Finally, without baseline data providing a clear indication of the situation at the time of the program’s inception, it is difficult to determine the extent to which the program has been successful in achieving its outcomes. Thus, it is quite difficult to determine the extent to which the program has been successful in achieving its immediate and intermediate outcomes as identified in the IASPP logic model.

There have also been challenges pertaining to reporting based upon the performance management plan outlined in the IASPP RMAF. The plan called for a report, which was to include a summary of interim reports for ongoing projects, as well as a summary by strategic priority and by habitat. This information is provided annually after the results of each request for proposals are determined in terms of which projects are to be funded. Furthermore, each individual proponent was to provide a project-end “Key Indicators of Success Report.” In total, 71 projects out of the total of 108 approved projects are complete and have submitted this report thus far. These reports were then to be consolidated to provide an overall analysis of the level of success of the program in terms of achieving its intended results. While that information has recently been consolidated, the program has not undertaken any analysis of the information.

Although data are being collected, the reporting requirements, as per the RMAF, are not being met. The number of performance indicators (114) as well as their diverse nature (i.e., environmental, economic, community capacity, outreach, innovation, engaging Canadians, training) has made it difficult for the program to consolidate performance data and to use this data to report on the overall success of the program in achieving its outcomes. Analysis of performance information has been a challenge in the face of managing the day-to-day delivery of the program. Therefore, performance data are not currently being used to inform program decision-making.

The results from the survey of proponents (Table 16) reveal that, for the most part, performance measurement and reporting requirements were articulated in the contribution agreements. Contribution agreements are generally very similar in their content. What may vary are the specific activities, budgets and expected results. The fact that not all proponents responded that expected results (74%) and reporting requirements (83%) were clearly articulated in the contribution agreement raises some questions as to whether the contribution agreements are consistently including all these items, or whether the respondent was not familiar with the terminology or contents of the contribution agreements themselves.

Table 16 Measuring Results
Performance ReportingSuccessful
YNDK
1. Was an approach to measure the expected results of your project clearly articulated in the contribution agreement?17

74%
33
2. Were reporting requirements established for your project?21

91%
11
3. Were reporting requirements clearly articulated in the contribution agreement?19

83%
22

Interviews with program staff revealed that they are aware of the deficiency in collecting and analysing performance information and hope that further automation of the system, including the development of an online reporting system to match the online application system, will assist in improving data collection and analysis.

Evaluation Issue: Design and Delivery

Q10: Are there appropriate governance structures in place to meet the objectives?

Indicator(s)
Methods
Rating
  • Established governance structure
  • Roles, responsibilities and accountabilities are well defined, well communicated, and understood.
  • Roles, responsibilities and accountabilities are functioning as intended.
  • Document review
  • File review
  • Interviews
Achieved

Governance structures are in place to meet the objectives of the IASPP. A Steering Committee, comprised of the Deputy Ministers of Environment Canada and Natural Resources Canada as well as the President of the Canadian Food Inspection Agency, has been established, and the roles, responsibilities and accountabilities for the Steering Committee are defined in the Interdepartmental Memorandum of Understanding between Environment Canada, Canadian Food Inspection Agency and Fisheries and Oceans Canada. Furthermore, an interdepartmental Director General Committee coordinates the overall implementation of the IASSC including the IASPP. A Technical Review Committee, comprised of experts from the three signatory departments, is responsible for reviewing and assessing project proposals. The Terms of Reference of the committee were established in the Interdepartmental Memorandum of Understanding between Environment Canada, Canadian Food Inspection Agency and Fisheries and Oceans Canada.

Interviews with senior management indicated that the governance structure appeared adequate. Interviews with program staff also revealed that there is clarity regarding the roles, responsibilities and accountabilities of the program. There was some acknowledgement that, in terms of accountability, the review of project outputs could be strengthened in order to improve performance management and to utilize that information for decision-making. Interview responses from the other government departments were also similar; most felt that their roles and responsibilities were clear.

Furthermore, during interviews with proponents, most mentioned that their roles and responsibilities were also clear. Some mentioned that there could be a potential benefit of better coordination/participation with the provinces, for example, in identifying regional IAS priorities for funding. Exploring means by which the provinces could be integrated into the TRC were also mentioned, but only if they are not potential project proponents.

Evaluation Issue: Design and Delivery

Q11: Has there been an assessment and strategic use of lessons learned?

Indicator(s)
Methods
Rating
  • Identified lessons learned and best practices
  • Identifies strengths and weaknesses
  • Factors that contribute to / detract from the achievement of results
  • Document review
  • Interviews
  • Survey
Progress made, attention needed

There has been some assessment and use of lessons learned from an operational perspective. In response to proponent feedback on the application system, the application process was automated. There is a desire to continue to evolve the online system to include reporting as well as the automation of the contribution agreement development process. From a more strategic perspective, there are no documented lessons learned although, recently, a process has been put in place, and an inter-departmental working group formed to examine options for the future of IASPP within the context of renewal of the national strategy.

Both successful and unsuccessful proponents were asked to comment on what they perceived as the IASPP’s key strengths and weaknesses presented in Figures 2 and 3, respectively. Of the 37 survey respondents (successful n=25, unsuccessful n=12), 31 responded to the question on strengths and 33 responded to the question on weaknesses. Overall, the key program strengths are that it is recognized as an important funding source (48%), and that the program has worthwhile goals and objectives (39%). The key program weaknesses are its approval process (52%), insufficient funding for the program (27%) and funding delays (24%). In two cases, the level of funding was raised as a main contributing factor to ensuring / limiting the effectiveness of a program. These are consistent with earlier findings.

A key weakness mentioned in the interviews was that many quality proposals go unfunded as a result of the size of the IASPP fund and the geographic coverage required of the program. As presented in Table 2, the program has received an average of over $7 million in proposals each year and can fund only $1 million per year. On average, only 16% to 19% of proposals are approved. As one proponent commented in the survey, when asked about the weaknesses of the program, it is “extremely poorly funded and as a result, good proposals are not fully funded, if at all.”

Figure 2: Survey Response to Program Strengths (n=31)

Figure 2: Survey Response to Program Strengths

Figure 3: Survey Response to Program Weaknesses (n=33)

Figure 3: Survey Response to Program Weaknesses (n=33)

5.0 Conclusions

This section concludes on the findings of the evaluation.

The evaluation of the IASPP revealed that the program remains relevant given that there is a continued need for a contribution program to address IAS-related issues in Canada at the grassroots level within the context of the national strategy. There are however issues with respect to the appropriate role for the IASPP that need to be addressed. Specifically, there remain questions with respect to alignment of the IASPP to current government priorities, as well as the effectiveness of the program in achieving its overall objectives given issues related to a lack of concise and focused program priorities and limited program funding.

Although the IASPP was well-aligned with government-wide priorities in 2005, when the program was created, there is only an indirect link between IASPP objectives and current government-wide priorities. With respect to departmental priorities, the evaluation found that the IASPP remains consistent with Environment Canada’s mandate, most importantly the protection of biodiversity.

The IASPP does contribute to some extent to the national strategy and complements the activities of other federal department and agency partners which primarily focus on science-based regulatory initiatives and national surveillance. The IASPP, on the other hand, focuses on promoting public compliance with regulatory approaches and establishing voluntary initiatives to address pathways not amenable to regulation. Environment Canada is also perceived as the best situated department to administer the IASPP and coordinate the national strategy, given that it is seen as a neutral party without a vested interest with respect to IAS.

The overall objectives of the program were found to be too broad, resulting in the program being spread too thinly across the country. According to the evidence collected, although the individual projects funded by the program were generally successful in achieving their respective objectives, these results were at the local/community level. As a result, program contributions have resulted in marginal gains, not major impacts at the national level. Most interviewees felt that there is a need to determine a concise and more focused set of key priorities for the program, where it could be expected to be able to make a difference.

The evaluation also found that the overall effectiveness of the program was hampered by the total amount of program funding. According to the evidence collected, the program could not achieve its overall objectives given limited resources available. Although the relevance of a contribution program was not questioned in and of itself, the ability of the IASPP to achieve its overall objectives given present funding levels was raised as a concern. Given this concern, some interviewees felt that diverting the funding to other programs under the national strategy could have a greater impact on addressing IAS-related concerns nationally. As a result, there is a need to examine the appropriate role for the IASPP, along with commensurate funding needed to effectively carry out this role for the program.

The evaluation also found that there are a number of areas that need to be addressed in the delivery of the program, including the implementation of an effective performance measurement strategy and the need to clearly communicate key priorities and objectives to potential proponents.

6.0 Recommendations

These recommendations are directed to the ADM, ESB.

Recommendations 1 and 2:

Overall, although there is a continued need for a contribution program in Canada to address IAS at the grassroots level, interviewees questioned the effectiveness of the IASPP to achieve its overall objectives given issues related to a lack of concise and focused program priorities and limited program funding. Most felt that although the individual projects funded by the program were generally successful in achieving their respective objectives, these results were at the local/community level. It was therefore felt that the contributions of the program resulted in marginal gains at the local/community level and not major impacts towards addressing IAS-related concerns at the national level. Some interviewees felt that given these results, the limited funds could be diverted to other programs under the national strategy to provide a greater impact at addressing IAS-related concerns at the national level.

Interviewees spoke of the need to re-examine the overall objectives of the program, in order to determine a few key priorities for the program where it could be expected to make a difference. At present, it was felt that the current objectives were too broad, resulting in the program being spread too thinly across the country, providing nominal funding to small localized projects, instead of focusing on a few key priorities that would result in the program making a bigger impact at addressing IAS-related concerns nationally.

Furthermore, limited program funding also had a negative impact on the overall effectiveness of the program to achieve its objectives, as it affected both the management of IASPP contribution agreements and Environment Canada’s ability to fulfil its coordinating role with respect to the national strategy. Many identified the need for a more effective coordination of the national strategy, something they felt was lacking at present.

Recommendation 1:

In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise with counterparts in participating departments and agencies to determine the appropriate role for the IASPP within the context of both the IASSC and EC departmental priorities, along with commensurate funding for the IASPP.

Recommendation 2:

Based on the decision taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, it is recommended that the program’s objectives be re-examined with an aim to develop a concise and more focused set of key priorities for the program to ensure that the IASPP contributes in an effective manner to both the national strategy and EC departmental priorities.

Recommendation 3:

Evidence collected as part of this evaluation illustrated a lack of clear understanding by project proponents about the overall objectives of the IASPP, thereby making it difficult for them to develop projects that link more directly with the program’s objectives. The application guidelines do provide some guidance, but it is at a very high level, with more emphasis on the evaluation criteria and listing examples of the types of activities that are funded, than specifically stating the objectives or the expected results of the IASPP.

Another key weakness of the IASPP lies in its limited ability to demonstrate achievement of its intended outcomes. This is due in part to the absence of a formal mechanism to validate the performance data that are collected by project proponents and provided to the program at the completion of individual projects. Thus, there is uncertainty about the accuracy of the performance data provided. Furthermore, the program presently collects information on approximately 114 key performance indicators, used to measure the level of achievement related to 56 project-level outputs and outcomes. The sheer number of performance indicators, as well as their diverse nature (i.e., environmental, economic, community capacity, outreach, innovation, engaging Canadians, training) has made it difficult for the program to consolidate performance data and use this data to report on the overall success of the program in achieving its outcomes. Furthermore, in some cases, there are issues with respect to the link between these performance indicators and the project-level outputs and outcomes that they are intended to measure. Thus, the program cannot use the indicators to determine whether IASPP-funded projects have been successful in achieving specific project results and overall program objectives.

As a result, it is quite difficult to determine the extent to which the program has been successful in achieving its immediate and intermediate outcomes, as identified in the IASPP logic model.

Based on the decision taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, as well as the articulation of program priorities as per Recommendation 2, it is recommended that:

a. the program clearly communicates its key priorities and objectives to potential project proponents to enable their understanding and thus ensure that projects which are developed are more directly linked to program objectives.

b. the program’s current performance measurement strategy be re-examined with the objective to develop and implement a set of intended project-level outcomes and performance indicators that would allow the program to be better able to demonstrate its results.

7.0 Management Response

The Assistant Deputy Minister, Environmental Stewardship Branch, Environment Canada (ADM, ESB) agrees with these three recommendations. Should there be a government decision to renew the IASPP, the program commits to the following actions in response to the three recommendations.

Recommendation 1:

In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise with counterparts in participating departments and agencies to determine the appropriate role for the IASPP within the context of both the IASSC and EC departmental priorities, along with commensurate funding for the IASPP.

If renewed, the specific role of and priorities for the IASPP should focus on the IASCC and EC departmental priorities, the priorities of the Canadian Food Inspection Agency (CFIA) and Fisheries and Oceans Canada (DFO) with whom EC shares governance of the IASPP, and the advice of other involved federal departments and agencies. The following action will be taken by the ADM, ESB to ensure this recommendation is addressed:

December 2009

  • ADM, ESB to discuss role of, priorities and commensurate funding for the IASPP in implementing the renewed IASSC with other federal departments and agencies.

Recommendation 2:

Based on the decision to be taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, it is recommended that the program’s objectives be re-examined with an aim to develop a concise and more focused set of key priorities for the program to ensure that the IASPP contributes in an effective manner to both the national strategy and EC departmental priorities.

Following confirmation of the role and commensurate funding for the renewed IASPP, the program’s objectives will be re-examined to create a more focused set of key priorities aligned with both the national strategy and EC departmental priorities. In doing so, EC will consult with the Canadian Food Inspection Agency, the Department of Fisheries and Oceans and other key involved federal departments and agencies with respect to their priorities for IASPP.

Specific actions to be taken to address this recommendation include:

December 2009

  • Subject to renewal of the IASPP, identify strategic outcomes, objectives and key priorities for the renewed IASPP in consultation with other federal departments and agencies.
  • Program to return to Environmental Sustainability Board with recommendations for future implementation of the IASPP.

Recommendation 3:

Based on the decision to be taken with respect to the appropriate role and commensurate funding of the IASPP as per Recommendation 1, as well as the articulation of program priorities as per Recommendation 2, it is recommended that:

a. the program clearly communicates its key priorities and objectives to potential project proponents to enable their understanding and thus ensure that projects which are developed are more directly linked to program objectives.

Subject to a renewed IASPP, EC will refine and update the IASPP application guidelines including project selection criteria to ensure greater clarity about the overall objectives of the IASPP, thereby ensuring projects link directly with the Program’s objectives.

Specific actions to be taken to address this recommendation include:

January 2010

  • Refine the application guidelines in line with revised Program objectives.
  • Provide guidance such as a list of eligible and ineligible projects to project proponents to ensure clarity with respect to IASPP objectives and priorities.

b. the program’s current performance measurement strategy be re-examined with the objective to develop and implement a set of intended project-level outcomes and performance indicators that would allow the program to be better able to demonstrate its results.

The performance measurement strategy of a renewed IASPP will be revised in line with the objectives and priorities to be established for the Program. Performance indicators will be streamlined to ensure collection and consolidation of meaningful data which can be used to report on the overall success of the program in achieving its outcomes.

To meet this goal, the following specific actions will be taken.

March 2010

  • Refine current logic model to reflect the new objectives (outcomes) for the Program
  • Complete a Performance Measurement Strategy with the aim to develop performance indicators most relevant to program objectives

March 2011

  • Improve the reporting format, implement an appropriate reporting tool and make other program adjustments as determined by the departmental G&C risk assessment strategy developed as part of the Departmental Action Plan for G&C Reform (CAPE Optimization Initiative)

March 2012

  • Adopt the departmental online application and information management system, which will enhance client service and facilitate application, monitoring and reporting processes for both clients and program staff (CAPE Optimization Initiative)

Contact persons: Bob McLean, Elizabeth Roberts

Annex A - National Invasive Alien Species Strategy for Canada Logic Model

Annex A -      National Invasive Alien Species Strategy for   Canada Logic Model

Click to enlarge

Annex B - Evaluation Matrix

Evaluation Issue: Relevance

Q1: Does the IASPP continue to be consistent with government-wide priorities and the departmental mandate?

Indicators
Data Source
Methodology
  • Program objectives that are aligned to government-wide priorities and departmental mandate.
  • Inter/intra-governmental relations and coordination (F/P/T and First Nations).
  • Invasive Alien Species Strategy for Canada, RPP, DPR, Speech from Throne
  • Terms of Reference and records of decision from coordinating bodies, MoUs, other agreements
  • Program staff
  • Document review
  • Interviews

Q2: Is there a continued need for the IASPP?

Indicators
Data Source
Methodology
  • Utility/rationale for the program
  • Ongoing need for capacity or activity in the area of departmental jurisdiction
  • Professional journals, papers
  • Environmental groups and other stakeholders
  • Document review
  • Interviews
  • Survey

Evaluation Issue: Success

Q3: Are planned activities being implemented and producing expected outputs?

Indicator
Data Source
Methodology
  • As per indicators in the performance measurement plan and reporting strategy
  • Program management documentation
  • MoUs, Agreements, Terms of References, Records
  • Project files
  • Stakeholder interviews
  • Survey
  • Document review
  • Interviews
  • Survey

Q4: Are immediate and intermediate outcomes being achieved as a result of the program?

Indicator
Data Source
Methodology
  • As per indicators in the performance measurement plan and reporting strategy
  • Project files and reports
  • Stakeholder interviews
  • Survey
  • Document review
  • File review
  • Interviews

Q5: Have there been any unexpected outcomes (positive or negative)? Were any actions taken as a result of these?

Indicators
Data Source
Methodology
  • Documented unexpected outcomes
  • Lessons learned or best practices have been documented
  • Project files and reports
  • Program documentation
  • Program staff and stakeholders
  • Document review
  • File review
  • Interviews

Evaluation Issue: Cost effectiveness/alternatives

Q6: Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?

Indicators
Data Source
Methodology
  • Financial management system lined to results management process
  • Consideration of alternate approaches
  • Financial management system
  • Finance department staff
  • Document review
  • Interviews
  • Survey

Evaluation Issue: Design and Delivery

Q7: Are program deliverables and expected outcomes identified clearly, and is the program being delivered as designed?

Indicators
Data Source
Methodology
  • Demonstration of the program’s expected deliverables and results
  • Consistency with documented design and delivery/implementation
  • Program documentation (RMAF, NIAS, etc.)
  • Program staff and stakeholders
  • Document review
  • Interviews
  • Survey

Q8: Are activities and outputs linked to shared outcomes and the overall government-wide objectives?

Indicator
Data Source
Methodology
  • Links between activities, outputs and outcomes supporting joint national objectives
  • Program documentation
  • Program staff and stakeholders
  • Document review
  • Interviews

Q9: Are performance data collected against program activities/outcomes and used for informed decision making?

Indicators
Data Source
Methodology
  • Measuring and reporting strategies and procedures
  • Presence of populated performance data system
  • Decisions based on performance information
  • Program Management System
  • Program staff
  • Document review
  • File review
  • Interviews
  • Survey

Q10: Are there appropriate governance structures in place to meet the objectives?

Indicators
Data Source
Methodology
  • Established governance structure
  • Roles, responsibilities and accountabilities are well defined, well communicated, and understood.
  • Roles, responsibilities and accountabilities are functioning as intended.
  • Program documentation, MoUs, Agreements, ToRs
  • Program staff and stakeholders
  • Document review
  • File review
  • Interviews

Q11: Has there been an assessment and strategic use of lessons learned?

Indicators
Data Source
Methodology
  • Identified lessons learned and best practices
  • Identifies strengths and weaknesses
  • Factors that contribute to / detract from the achievement of results
  • Program documentation
  • Program staff
  • Document review
  • Interviews
  • Survey


Annex C - List of Background Information and Supporting Documentation

Document/Files

  1. List of Funded IASPP Projects – 05-06
  2. List of Funded IASPP Projects – 06-07
  3. List of Funded IASPP Projects – 05-06
  4. Treasury Board Submission – Implementing an Invasive Alien Species Strategy for Canada
  5. EC IASPP RMAF March 2008
  6. RMAF – Invasive Alien Species Strategy for Canada May 2008
  7. An Invasive Alien Species Strategy for Canada
  8. IASPP Online Application Process – Survey Results
  9. IASPP Summary Data (Revised Dec 16 - summary of projects)
  10. National Aquatics Invasive Species Committee – Terms of Reference
  11. MOU IAS Partnership Program
  12. Contribution Agreement Development Process and Tips
  13. EC Common Indicators – Grants and Contributions
  14. Memo to DM – call 2006-07 and annexes
  15. Memo to DM – call 2007-08 and annexes
  16. Memo to DM – call 2008-09 and annexes
  17. Technical Review Committee (TRC) Terms of Reference
  18. IASPP Project Performance Summary – updated January 9, 2009
  19. IASPP Successful Applicants (Call 1-4)
  20. IASPP Unsuccessful Applicants (Call 4)
  21. 2008 TRC meeting summary data
  22. Evaluation Summary 2008 TRC version
  23. IASPP Minutes and Comments
  24. Report IASPP Technical Review Committee Meeting May 3-4 2007
  25. Report IASPP Technical Review Committee Meeting March 08
  26. Technical Review Committee – 0708 selected projects
  27. IASPP Summary Data – revised December 19
  28. Financial reports
  29. Environment Canada RPP 2008-2009
  30. Environment Canada DPR 2007-2008
  31. Project files (8)
  32. IASPP Application Guidelines September 2008

Annex D - List of Interviewees

Key Informant Interviews - Round I
DepartmentNumber of Regions RepresentedNumber of Individuals Interviewed
Environment Canada18
Other Government Departments
CFIA; NRCan; DFO; AAFC15
Other Stakeholders
Provincial33
Non-Governmental Organizations33
Total 19
Key Informant Interviews - Round II
DepartmentNumber of Regions RepresentedNumber of Individuals Interviewed
Environment Canada program staff and senior management14
Interdepartmental Directors General IAS Steering Committee members24
Other Federal Government Departments and Agencies
CFIA; NRCan; AAFC14
Total 12


Annex E - Summary of Findings19

Evaluation Question (EQ)Rating
Relevance:
Q1: Does the IASPP continue to be consistent with government-wide priorities and the departmental mandate?Progress Made, Attention Needed
Q2: Is there a continued need for the Department’s contribution?Progress Made, Attention Needed
Success:
Q3: Are planned activities being implemented and producing expected outputs?Progress Made, Attention Needed
Q4: Are immediate and intermediate outcomes being achieved as a result of the program?Progress Made, Attention Needed (~)
Q5: Have there been any unexpected outcomes (positive or negative)? Were any actions taken as a result of these?Not Applicable
Cost Effectiveness:
Q6: Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?Achieved
Design and Delivery:
Q7: Are program deliverables and expected outcomes identified clearly, and is the program being delivered as designed?Progress Made, Attention Needed
Q8: Are activities and outputs linked to shared outcomes and the overall government-wide objectives?Progress Made, Attention Needed
Q9: Are performance data collected against program activities/outcomes and used for informed decision making?Little Progress, Priority for Attention
Q10: Are there appropriate governance structures in place to meet the objectives?Achieved
Q11: Has there been an assessment and strategic use of lessons learned?Progress Made, Attention Needed



1 Environment Canada. Invasive Alien Species Partnership Program website.

2 The Invasive Alien Species Strategy for Canada (2004) defines pathways as the “different ways that IAS can be introduced” or “pathways of invasion” which may include transportation vectors (e.g., boats, trains, planes) or in packaging material.

3 The design for the IASPP evaluation was carried out in the 2008-09 fiscal year, prior to implementation of the new Treasury Board Policy on Evaluation. The current evaluation reflects those issues outlined in the 2001 evaluation policy that was in effect at the time this evaluation was conducted.

4 Given that the information gathered to address this question is similar in nature to the information gathered to address Question 7 under Design and Delivery, the decision was taken to present the findings for both these questions together, under Question 7.

5 Government of Canada. An Invasive Alien Species Strategy for Canada. September 2004. Pg. 10.

6 Government of Canada. An Invasive Alien Species Strategy for Canada. September 2004. Pg. 9.

7 Plant and plant pests are regulated under the Plant Protection Act and Animal Quarantine Act.

8 Invasive alien species in Canadian environments.

9 The International Union for Conservation of Nature is one of the largest global environmental networks, with more than 1,000 government and NGO member organizations, and almost 11,000 volunteer scientists in more than 160 countries.

10 Growing Forward focuses on building a profitable agricultural sector through its three strategic outcomes: a competitive and innovative sector, a sector that contributes to society’s priorities, and a sector proactive in managing risks).

11 Regional/provincial councils generally work in a specific region or province to minimize the negative ecological, social, and economic impacts caused by the introduction, establishment, and spread of invasive plants. They are non-profit organizations whose members are involved in all aspects of invasive plant management. Members may include technical specialists working for government and industry, weed committee coordinators, First Nations representatives, foresters, forest technologists, biologists, ranchers, horticulturists, recreation enthusiasts, gardeners, and other concerned individuals.

12 Partners in this instance refer to the actual project proponents.

13 Performance data are presented in Tables 5, 6, 7, 8, 9, 10, 11 and 12.

14 For the purposes of this evaluation, jurisdictional coordination was defined as inter-departmental coordination at the federal level, federal and provincial/territorial coordination, municipal/community coordination, or any mix thereof.

15 Contribution expenses in fiscal years 2007-08 and 2008-09 include subsequent payments of multi-year contribution agreements approved in fiscal years 2005-06 and 2006-07.

16 Participants included provincial/territorial and federal government representatives, as well as representatives of jurisdictional IAS councils.

17 Eligibility requirements are both project-specific (e.g., must be less than $50,000, cannot be on federal land) and proponent-specific (e.g., organizations can be non-governmental organizations, provincial, territorial or municipal crown agencies).

18 Examples of selection criteria are technical feasibility, proponent capacity, public education value, and risk reduction value.

Date modified: