This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Evaluation of the National Agri-Environmental Standards Initiative

Final Report

July 20, 2009

( Word Version , 1.60 Mb)

Report Clearance Steps

Planning phase completed
March 2008
Report completed
May 2009
Report approved by Departmental Evaluation Committee
July 2009


Acronyms Used in the Report

AAFC
Agriculture and Agri-Food Canada
ADM
Assistant Deputy Minister 
APF
Agricultural Policy Framework
APS
Achievable Performance Standards
BMPs
Beneficial management practices
BPC
Building Public Confidence
CB-IPS
commodity-based ideal performance standards
CCME
Canadian Council of Ministers of the Environment
CEFI
Canadian Ecological Flow Index 
DG
Director General
DM
Deputy Minister 
EC
Environment Canada
ENGO
Environmental non-governmental organization
IPS
Ideal Performance Standards
MOU
Memorandum of Understanding
NAESI
National Agri-Environmental Standards Initiative
NAHARP
National Agri-Environmental Health Analysis and Reporting Program
PMRA
Pest Management Regulatory Agency
RMAF/RBAF
Results-Based Management and Accountability Framework/Risk-Based Audit Framework
RPP
Report on Plans and Priorities
TBS
Treasury Board of Canada Secretariat
TSS
Total Suspended Sediments 
WEBs
Watershed Evaluation of Beneficial Management Practices 

 


Acknowledgements

The Evaluation Project Team, led by Michael Callahan under the direction of Shelley Borys, would like to thank those individuals who contributed to this project, particularly all the interviewees, who provided crucial insights and comments. We would also like to thank the members of the Joint Evaluation Committee for their guidance and support: Elizabeth Roberts, from Environment Canada, and Rohit Tuteja and Robin MacKay, from Agriculture and Agri-Food Canada.

The evaluation was conducted by PRA Inc., which prepared this report along with the Evaluation Division, Audit and Evaluation Branch, Environment Canada.

Contents

Executive Summary

1.0 Introduction

1.1 Definition of Standards

1.2 Outline of the Report

2.0 Overview of the Initiative

2.1 Background

2.2 Objectives and Expected Results

2.3 Financial Resources

2.4 Initiative Reach – Stakeholders and Partner

2.5 Governance Structure

2.6 Roles and Responsibilities

2.7 Performance Reporting

2.8 Initiative Outputs and Outcomes

3.0 Evaluation Design

3.1 Purpose and Scope of Evaluation

3.2 Evaluation Approach and Methodology

3.3 Limitations of the Evaluation

3.4 Evaluation Ratings

4.0 Findings

4.1 Relevance

4.2 Design and Delivery

4.3 Success

4.4 Cost-Effectiveness

5.0 Conclusions

5.1 Main Findings

5.2 Key Lessons Learned

Appendix 1 Evaluation Issues and Questions

Appendix 2 Interview Guides

Appendix 3 References

Appendix 4 NAESICommunications Activities

Appendix 5 NAESITimeline

Appendix 6 NAESIStandards by Thematic Area

Appendix 7 NAESIOutputs

Appendix 8 Summary of Findings


Executive Summary

Introduction

Science plays two key roles in mitigating the external effects of farming. First, science identifies the limits of soil, water and air to absorb farming activity. Second, science provides the theoretical framework for designing and evaluating mitigating strategies, commonly termed beneficial management practices.

Led by Environment Canada, in partnership with Agriculture and Agri-Food Canada, the National Agri-Environmental Standards Initiative (NAESI) received $25 million through a Memorandum of Understanding between Agriculture and Agri-Food Canada and Treasury Board of Canada Secretariat for five years (2003–2004 to 2007–2008) to “develop national environmental performance standards for agricultural production” (Lefebvre 2006).1 Focusing on four thematic areas--water, biodiversity, pesticides and air--these standards are intended to support, qualitatively and quantitatively, the development of non-regulatory agri-environmental actions under the Agricultural Policy Framework and other initiatives by government and environmental non-governmental organizations.

Environment Canada held a final stakeholder meeting in January 2009 and will deliver these performance standards to Agriculture and Agri-Food Canada in 2009 for implementation. The NAESIMemorandum of Understanding required that an evaluation of the initiative be done.

The Evaluation Division, Audit and Evaluation Branch, of Environment Canada oversaw the evaluation. The Department commissioned PRA Inc., an independent research firm, to carry out the necessary data collection and analysis.

Evaluation Issues

The evaluation examined the following issues:

Relevance: Was the initiative consistent with departmental and government-wide priorities?

Design and delivery: Was the initiative designed and delivered in the best possible way?

Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?

Cost-effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?

Methodology

Since NAESIhas ended and is not being considered for renewal, the evaluation was limited in scale, relying on two qualitative lines of evidence--interviews and document review. 

  • The study team interviewed 18 representatives from Environment Canada, Agriculture and Agri-Food Canada, other federal government departments, provincial governments and other stakeholders (e.g. environmental non-governmental organizations and industry). 
  • The study team reviewed relevant documents and literature to compile a profile of the initiative and to answer specific evaluation questions. Environment Canada provided most of the documents for review.

Overview of Findings and Conclusions

NAESIaligned with Environment Canada’s strategic outcomes and program areas. It was also part of the Environment Pillar of Agriculture and Agri-Food Canada’s Agricultural Policy Framework and is logically consistent with Growing Forward, the successor to the to this framework. Interviewees said that the four thematic areas selected to categorize the standards were appropriate for agri-environmental programming.

The partnership between Environment Canada and Agriculture and Agri-Food Canada was seen as reflecting good design practice, as was the practice of involving stakeholders such as industry groups. All interviewees supported the governance structure and the appointment of a single designated program manager, seeing these as effective delivery processes.  

NAESIincreased scientific understanding of the relationship between agriculture and the environment, and developed 98 standards. While some variation exists in the number and completeness of the standards developed across the four thematic areas, and work remains to be completed in some areas, interviewees said that a significant body of research has been synthesized to support agri-environmental policy and programming.

The administrative costs of NAESIwere low, and the initiative did not duplicate existing programming in Canada. The initiative linked to important federal agri-environmental initiatives, and most interviewees said that the initiative generated state-of-the art scientific information; however, it is difficult to judge the ultimate value of NAESI, since it is unclear whether and how the standards developed will be implemented.

The realization of the long-term strategic outcomes of NAESI(e.g. improved stewardship by agricultural producers of land, water, air and biodiversity) will require the development of an explicit implementation and adoption strategy to integrate these standards into emerging agri-environmental programming and policy, especially within Growing Forward.

Findings by Evaluation Issue

Relevance: Was the initiative consistent with departmental and government-wide priorities?
  • NAESIwas consistent with departmental and government-wide priorities. It aligned with Environment Canada’s and Agriculture and Agri-Food Canada’s strategic outcomes, particularly those relating to air, water and soil quality, and biodiversity, and formed part of the Environment Pillar of the Agricultural Policy Framework. It also addressed the protection of the environment, which was one of the priorities the Government of Canada identified in the 2007 Speech from the Throne (Canada 2007a).
Design and delivery: Was the initiative designed and delivered in the best possible way?
  • Overall, NAESIwas well designed and delivered. The initiative focused on an appropriate set of thematic areas, its governance structure was effective, and the Memorandum of Understanding clearly defined the roles and responsibilities of Environment Canada and Agriculture and Agri-Food Canada. The partnership between these two departments was integral to the development of the agri-environmental standards, while the identification of a single program manager helped facilitate good communication.
  • A key weakness of NAESIwas the failure to clearly communicate to stakeholders and beneficiaries how the standards developed through the initiative would be used. In addition, the initiative did not define strategies for integrating or implementing the standards. 
Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?
  • The initiative was effective in achieving its immediate and intermediate outcomes and its ultimate outcomes. NAESIcontributed to the federal government’s understanding of the relationships and links between agriculture and the environment, expanded Agriculture and Agri-Food Canada’s knowledge base and increased Environment Canada’s understanding of agricultural production practices. NAESIgenerated the scientific research needed to develop standards and lay the foundation for the development of other standards. This work resulted in 98 standards, which can serve as benchmarks for environmental performance and be used to inform decision making. However, the completeness of the standards developed varies across themes. 
  • While NAESIwas completed on time and budget, it is too soon to draw conclusions about its progress toward its long-term strategic outcomes. An implementation plan was not established at the outset of the initiative. Further, at the time this report was prepared, there was no plan for the standards to be reflected in the programming or policies under Growing Forward, a key mechanism for implementation. It was also unclear how these standards might support provincial and territorial agri-environmental policies and programming.
Cost-effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?
  • NAESI’s administration costs were minimal, and the NAESIteam effectively collaborated with counterparts on other projects and initiatives. NAESIdid not duplicate other programs and had effective links with other programs, such as the National Agri-Environmental Health Analysis and Reporting Program, the Watershed Evaluation of Beneficial Management Practices, and the Building Public Confidence in Pesticide Regulation and Improving Access to Pest Management Initiative, and followed the guidelines of the Canadian Council of Ministers of the Environment. 
  • Despite being implemented in a cost-effective manner, it is difficult to judge whether NAESIwas good value for money. Implementation of the standards remains an open question; if the standards and guidance from NAESIwere to find expression in neither Growing Forward programming nor provincial or territorial agri-environmental programming, then the initiative would not have realized its intended long-term strategic outcomes, which would limit the value of the initiative for the federal dollars invested.

Lessons Learned

Since NAESIhas ended, the focus of this evaluation was on lessons learned as opposed to recommendations for improvement. The following lessons learned from the experience with NAESImay serve to inform the design of future similar initiatives.

  1. Clearly stating the intent and objectives of the initiative at its outset, especially with respect to implementation, increases partner and stakeholder support and buy-in. This also involves ensuring all stakeholders understand the language used in memoranda of understanding and communication materials. Had this happened for this initiative, work on the incomplete standards may have increased and stakeholder implementation of NAESI’s outputs may have been encouraged.
  2. A single program manager and point of contact proved instrumental to this horizontal initiative. The manager facilitated interdepartmental communication and coordinated diverse technical initiatives.
  3. Maintaining strong relationships and communication with partners is essential not only to the creation of an initiative’s intended outputs but also for the realization of its intended outcomes. For an initiative such as NAESI, the standards developed must be integrated into agri-environmental programming that supports on-farm adoption of beneficial management practices. Agriculture and Agri-Food Canada has primary responsibility for ensuring that NAESIoutputs become integrated into the agri-environmental programming under Growing Forward. 
  4. The primary objective of NAESIwas to develop agri-environmental standards. While NAESIdeveloped national standards, synchronization with existing or planned provincial standards or regulations remains unfinished. Consultation with provincial stakeholders at an earlier stage of the initiative might have enabled NAESIto become integrated with provincial processes. This might also have helped stimulate additional support for the initiative. 
  5. Defining an implementation strategy for the outputs is essential for their integration into agri-environmental programming. Although NAESIadded to the scientific knowledge base and established standards, it lacked an implementation strategy; therefore, it cannot be said that it has progressed toward its strategic outcomes. In the final analysis, realization of the strategic outcomes will require an implementation plan to promote the use and application of the standards through the agri-environmental programming under Growing Forward.

Top of Page

1.0 Introduction

Science plays two key roles in mitigating the effects of farming. First, it identifies the limits of soil, water and air to absorb farming activity. Second, it provides the theoretical framework for designing and evaluating mitigating strategies, commonly termed beneficial management practices.

Agriculture and Agri-Food Canada’s Agricultural Policy Framework supported concrete improvement in the quality of the environment through the coordinated adoption of beneficial management practices on farms. This framework aimed to increase and improve the use of environmental planning tools and management systems by Canadian farmers.

Approved in February 2003, the Agricultural Policy Framework set out the need to develop a suite of non-regulatory standards for the environmental quality required of agriculture and to validate beneficial management practices. Non-regulatory standards promote environmental practices that reduce agricultural risks, thus benefiting the health and supply of water, the health of soils, the health of air, and the compatibility of biodiversity and agriculture. Non-regulatory standards also provide a benchmark for reporting to Canadians on the results of improved agricultural management practices. 

The National Agri-Environmental Standards Initiative (NAESI), led by Environment Canada in partnership with Agriculture and Agri-Food Canada, supported the role of science in mitigating the external effects of agriculture. NAESIhad the following goals:

  • to establish non-regulatory national environmental performance standards (with regional application) that define common Environment Canada and Agriculture and Agri-Food Canada goals for the environment;
  • to evaluate standards attainable by use of existing environmentally beneficial agricultural production and management practices; and
  • to increase understanding of relationships between agriculture and the environment (Agriculture and Agri-Food Canada, n.d.). 

This initiative received $25 million through a Memorandum of Understanding between Agriculture and Agri-Food Canada and Treasury Board of Canada Secretariat for five years (2003–2004 to 2007–2008).2 Over the five years, NAESIused this funding to develop environmental performance standards to guide beneficial management practices to be implemented at the farm level. Utilizing funds carried over from 2007–2008, Environment Canada held a final stakeholder meeting in January 2009 to review the performance standards, which it will deliver to Agriculture and Agri-Food Canada in 2009 for implementation. The Memorandum of Understanding required an evaluation of NAESI.

This evaluation assessed the relevance of NAESIand whether and to what extent it achieved its intended outcomes, as well as whether it was designed and delivered in the most appropriate and cost-effective way. Lessons learned from the initiative could serve to inform the design of future similar initiatives. 

A committee of program and evaluation representatives from Environment Canada and Agriculture and Agri-Food Canada guided the planning phase of the evaluation, including developing the interview questions and a list of interviewees. Environment Canada’s Evaluation Division, Audit and Evaluation Branch, oversaw the evaluation and commissioned PRA Inc., an independent research firm, to conduct the required interviews and analysis.

Top of Page

1.1 Definition of Standards

NAESIdeveloped agri-environmental standards in four areas:

  • water (aquatic ecosystem health): nutrients, pathogens and instream needs, as well as sediments and water availability;
  • biodiversity (aquatic and terrestrial ecosystem health): habitat conservation, including of wetlands, riparian areas and connective corridors;
  • pesticides (aquatic and terrestrial ecosystem health): 20 priority pesticides; and 
  • air (human health): ammonia emissions. 

Within these four areas, there are two types of standards: ideal performance standards, which “specify the desired level of environmental state needed to maintain ecosystem health,” and achievable performance standards, which “specify the level of environmental quality that can realistically be achieved using currently available and recommended beneficial management practices” (Roberts 2009). Specifically, these standards comprise the following:  

  • non-regulatory instruments;
  • quantitative or qualitative measures (descriptive benchmarks) of desired environmental qualities that are scientifically defensible and focused on key agri-environmental issues;
  • useful tools to inform landowners, decision makers and policy-makers when setting targets for desired degrees of environmental performance for air and water quality, biodiversity and pesticide use in agricultural areas; and
  • definitions of maximum concentrations of harmful substances, specified condition of the environment or habitat standards (biodiversity) (Canada 2007b).

Top of Page

1.2 Outline of the Report

This document presents the findings and conclusions of the evaluation. Section 2 provides background information on NAESI. Section 3 describes the purpose of this evaluation and the methodology used. Section 4 presents the evaluation’s findings. Finally, Section 5presents the conclusions and lessons learned based on the evaluation findings.

Top of Page

2.0 Overview of the Initiative

The purpose of NAESI, which fit under the Environment Pillar of the Agricultural Policy Framework, was to develop “national non-regulatory, science-based agri-environmental performance standards […] to guide agri-environmental decision-making.”3 Environment Canada and Agriculture and Agri-Food Canada signed a Memorandum of Understanding in December 2003 to implement NAESI. Work commenced April 1, 2004, with the program scheduled to end March 31, 2008. (In actual fact, work continued past that date to ensure all products were completed.)

NAESIhad two activity areas (see “2.8 Initiative Output and Outcomes,” below). 

  • Activity area 1 assessed “risk, priorities, and partnerships for performance standards” for the thematic areas.  
  • Activity area 2 developed national standards intended to reduce the impact of the agricultural sector on the environment. 

Work on NAESIwas undertaken in stages: Stage 1 (2004–2005) was a scoping exercise to determine those aspects of the environment most affected by agriculture. This first stage supported priority setting, which preceded the development, demonstration and testing of standards in Stage 2 (2005–2008). Stage 3, which took place during the last year of the initiative (2007–2008), involved the conclusion of standards testing and the delivery of results to Agriculture and Agri-Food Canada (Roberts 2006; Bowerman et al. 2009, sec. 1.4). The tangible outputs of the initiative are standards compiled in a series of synthesis reports.

NAESIused a peer review process to review the standards as expressed in the synthesis reports. The peer review process included an academic review overseen by an independent evaluator, followed by a technical review by experts from partner and stakeholder organizations. The process assessed the methodologies used to develop the standards and the standards themselves. In effect, this review process assessed “the science behind standard development rather than issues of policy and implementation” (Bowerman et al. 2009, sec. 2.4). The peer review served as a check on the scientific validity and quality of the outputs.

Top of Page

2.1 Background

NAESIwas one of several agricultural programs outlined in the Agricultural Policy Framework, a five-year (2003–2008), $5.2-billion initiative of the Government of Canada and the provincial governments. (The Framework was replaced by Growing Forward; see “4.1 Relevance,” below.) The Framework consisted of five integrated pillars for programming and policy development: Business Risk Management, Food Safety and Quality, Science and Innovation, Environment, and Renewal. NAESIfell under the Environment Pillar (Agriculture and Agri-Food Canada 2003). Policy and programs under the Environment Pillar were allotted $526 million to improve agricultural producers’ stewardship of the soil, water and air, and enhance biodiversity. To this end, the Government of Canada and the provinces agreed to conduct research and development efforts intended to do the following: 

  • increase understanding of relationships between agriculture and the environment;
  • develop and evaluate environmentally beneficial agricultural production and management practices; and
  • establish agri-environmental standards that support the common goals for environment (Agriculture and Agri-Food Canada 2003, sec. 24, 26.3).

Top of Page

2.2 Objectives and Expected Results

The goals of NAESIwere stated in the Memorandum of Understanding and were also discussed in the Overarching Report (Bowerman et al. 2009). The goals included establishing non-regulatory environmental standards to support the common goals of the Environment Pillar of the Agricultural Policy Framework. The Memorandum of Understanding identified two types of standards:

  • performance standards, which identify a "desired level of environmental state to maintain ecosystem health" and specify the quality levels achievable through existing technology and practice; and
  • process standards, which identify the specific practices (beneficial management practices) that support the performance standards (Canada 2003).

Figure 1 shows the logic model for the initiative, depicting the results chain leading from activities to outputs to outcomes. Through NAESI, the Government of Canada undertook data collection, benchmarking and research to produce and disseminate information and guidance related to the four themes for agri-environmental standards: water, biodiversity, pesticides and air. The results of this work supported the realization of the following immediate and intermediate outcomes:

  • development of methods for ideal and achievable performance standards; 
  • effective technology transfer packages and communications support materials for a range of audiences (e.g. technical, policy-makers and stakeholders); and
  • increased understanding of the relationships and links between agriculture and the environment through the collection of scientific information.

The Results-Based Management and Accountability Framework/Risk-Based Audit Framework for NAESInoted that the ultimate strategic outcomes for the initiative, as set out in the logic model, were identified for purposes of the Environment Pillar of the Agricultural Policy Framework. They included improved stewardship of the environment by producers and increased confidence that food is being grown safely and in an environmentally responsible way (Environment Canada 2005, sec. 2.6). 

It is important to note that, with its four years of funding, NAESIwas only intended to produce standards; therefore, its ultimate outcome was the creation of these standards. The strategic outcomes identified in the logic model clearly will require extensions to the activities and resources made available for NAESI.

Figure 1. Logic Model 2.3 Financial Resources

Figure 1.  Logic Model (view large image)

Top of Page

2.3 Financial Resources

The Memorandum of Understanding and the Results-Based Management and Accountability Framework/Risk-Based Audit Framework stipulated the transfer of $25 million to Environment Canada over five years, starting on April 1, 2003. The profile of funding for NAESIchanged slightly between the initial conception of the program and the late signing of the Memorandum of Understanding in December 2003, which delayed the start of NAESIuntil April 1, 2004 (Bowerman et al. 2009). The new allocation was based on the recommendation of the ADM Joint Management Committee (see Section 2.5) of May 13, 2004 (Environment Canada 2005, sec. 3.0). Table 1 shows the allocation of funds, broken down by fiscal year and activity area, as discussed in Section 2.

 

Table 1: Allocations by fiscal year and activity area. 2003–2008
 Allocations (millions)
2003–20042004–20052005–20062006–20072007–20082003–2008
Activity area 1: Assess risk priorities and partnerships for performance standard development$1.21$0.99$2.20
Activity area 2: Develop agri-environmental performance standards$4.39$4.71$5.70$4.70$19.50
Program delivery and regional coordination$0.90$0.80$0.80$0.80$3.30
Total$6.50$6.50$6.50$5.50$25
Original allocation$2.30$6.72$6.20$5.50$4.28$25

Source: (Bowerman et al. 2009, pp. 185–186) 

As Table 1 shows, funding for activity area 1, which mostly involved planning, was restricted to the first two years of the initiative. The funding amounted to $2.2 million, or 9 percent of the total NAESIbudget. Program delivery and regional coordination accounted for an additional $3.3 million between 2004 and 2008, or about 13 percent. The remaining 78 percent of the funding ($19.5 million) was allocated to activity area 2. Table 1 also shows how the $2.3 million originally set to flow to Environment Canada in 2003–2004 was re-allocated to support program activities in 2004–2008. 

Top of Page

2.4 Initiative Reach: Stakeholders and Partners

Numerous partners were associated with NAESI, including Agriculture and Agri-Food Canada, Environment Canada and the Treasury Board of Canada Secretariat, as well as researchers, contractors, universities and other institutions involved in developing the standards and associated guidance material. Researchers from environmental non-governmental organizations and provincial governments involved in parallel projects also collaborated on the completion of certain NAESIstandards.

The intended beneficiaries of NAESIincluded the agriculture and agri-food sector, the Canadian public and global consumers, who collectively represented NAESI’s target population. The agricultural sector was expected to benefit from information that supports improved agricultural practices that increase sustainability, a better reputation for environmental responsibility and increased ability to respond to “green market opportunities,” as well as continued access to international markets (Roberts 2006; Environment Canada 2005, sec. 3.0). In the long term, it was expected that Canadian and global consumers of Canadian agricultural products would benefit from increased confidence that these products were produced in an environmentally responsible way. NAESIwas also expected to benefit the Canadian economy by “improving understanding of environmental costs and benefits for agriculture,” as well as the Canadian public by contributing to “a healthier environment, better quality of life, and sustainable management of natural resources for future generations” (Bowerman et al. 2009, sec. 1.5).

Top of Page

2.5 Governance Structure

NAESI’s governance structure consisted of three levels of management committees. Representatives from each department populated the Deputy Ministers (DM) Committee, the Assistant Deputy Ministers (ADM) Joint Management Committee and the Director General (DG) Steering Committee. The Interdepartmental DGSteering Committee was initially chaired by the Executive Director, Habitat Conservation and Protected Areas, Environmental Stewardship Branch, Environment Canada, and then by the Director General, Public and Resources Sectors Directorate, Environmental Stewardship Branch, Environment Canada. This committee reported to the ADM Joint Management Committee, which, in turn, reported to the DM Committee. The Treasury Board of Canada Secretariat was an ex-officio member on the DG Steering Committee and the ADM Joint Management Committee (Bowerman et al. 2009, p. 6). Four thematic teams, each headed by a theme lead, reported to the NAESISecretariat, which, in turn, regularly communicated with Agriculture and Agri-Food Canada. NAESI’s governance structure is illustrated in Figure 2. See “2.6 Roles and Responsibilities,” below, for information on the roles and responsibilities of each committee.

Figure 2. NAESI’s governance structure 

Figure 2. NAESI’s governance structure 

Source: Bowerman et al. 2009 

Top of Page

2.6 Roles and Responsibilities

The roles and responsibilities of Environment Canada and Agriculture and Agri-Food Canada included those related to the management of the Memorandum of Understanding (MOU) and those related to the delivery of NAESI

Agriculture and Agri-Food Canada reviewed and approved work plans, progress reports and reporting on the implementation of the MOU, in addition to providing funding for NAESI. This department also provided scientific and technical advice and information for other of its agri-environmental initiatives and offered policy advice when the objectives of the MOU changed. During the implementation of NAESI, Agriculture and Agri-Food Canada’s responsibilities included providing advice and guidance, coordinating committee participation, being involved in policy discussions related to standards development, facilitating stakeholder participation, and contributing of agricultural knowledge and expertise. 

Environment Canada led the five-year strategic planning, prepared quarterly progress reports and disseminated information on changes to the objectives of the MOU. Further, it managed the development and execution of annual work plans and oversaw expenditures. Environment Canada also identified priority areas for the development of agri-environmental standards and produced standards in these areas. Finally, this department drafted a series of annual reports and the final synthesis reports discussing the suite of performance standards, and provided recommendations for the use of the standards in the development of process standards by Agriculture and Agri-Food Canada. To carry out these responsibilities, Environment Canada consulted with stakeholders to assess priorities for standards development, and coordinated workshops and other communication activities to discuss and inform the framework for managing agri-environmental standards.

The ADM Joint Management Committee reported to the DMCommittee semi-annually and approved annual costs and work plans, reviewed quarterly results and cash flow forecasts and expenditures, approved small changes to the MOU, and recommended changes to the objectives of the MOU to the DG Committee. The DM Committee met annually to discuss the performance of NAESI, recommend to the Treasury Board of Canada Secretariat the amount of funding to be released, approve re-allocations of funds between fiscal years or departments, resolve disputes relating to the implementation of the MOU, and receive approval from the Minister of changes to the MOU’s objectives. The NAESISecretariat, based in Environment Canada, was responsible for day-to-day management of the activities under the four themes, specifically coordinating annual work plans and budget allotments, and communications activities (see Appendix 4). Each of these entities, as well as the Treasury Board of Canada Secretariat, reviewed the Results-Based Management Accountability Framework/Risk-Based Audit Framework, including the evaluation matrix and the performance measures and indicators.

Top of Page

2.7 Performance Reporting

The following was the reporting strategy for NAESI:

  • Environment Canada provided semi-annual reports on the progress of the initiative, using a standard template.
  • Environment Canada reported internally on its performance in regards to NAESIin the annual Departmental Performance Report as a mechanism for demonstrating accomplishments.
  • Agriculture and Agri-Food Canada used the information Environment Canada provided on this initiative to report on accomplishments in the context of the Agricultural Policy Framework.
  • Environment Canada made the final report on the NAESIevaluation available as a public document.

Top of Page

2.8 Initiative Outputs and Outcomes

In addition to the annual Technical Series consisting of a total of 247 reports over four years, NAESIculminated in the development of 16 synthesis reports--15 peer-reviewed technical reports and the Overarching Report. In total, 98 standards were developed and recommended across the four thematic areas. This sub-section discusses the three classes of standards, presents the scale of the standards, and reviews the thematic areas and the associated standards.

Classes of NAESIStandards

NAESI’s Results-Based Management and Accountability Framework/Risk-Based Audit Framework outlined two types of non-regulatory standards considered important for environmental performance: performance standards and process standards. Performance standards are “intended to identify the degree of desired environmental performance,” while process standards “recommend the methodology of production to achieve the performance standards.”4 Two types of performance standards were developed: ideal performance standards, which specify “the desired level of environmental state needed to maintain ecosystem health,” and achievable performance standards, which specify “the level of environmental quality that can be achieved using recommended, best available processes and technologies.”5 The ideal and achievable states of the environment, as dictated by the standards, need not be equal and may be expected to vary in many circumstances.

For example, Figure 3 depicts a situation that is common in many agricultural contexts. The figure shows that as the magnitude of an agricultural stressor increases, the impact on the environment also increases. In the “current situation,” the magnitude of the stressor is large, and, so, the corresponding environmental impact is high. The desired environmental state, as represented by ideal performance standards, would dictate much less of the stressor, resulting in a much lower impact. However, with available beneficial management practices, it is only feasible to lower but not eliminate the environmental impact.

Figure 3. One potential relationship between ideal and achievable performance standards

Figure 3. One potential relationship between ideal and achievable performance standards

Source: Bowerman et al. 2009, sec. 1.6

The Overarching Report noted that “employing different methodologies to develop the standards […] coupled with data availability issues and time constraints led the recommended standards developed through NAESIto vary in format” (Bowerman et al. 2009, sec 1.61). The work undertaken through NAESI, therefore, led to the development of two formats of ideal performance standards, as well as to the development of achievable performance standards. The former were numerical or descriptive benchmarks, and methodological support and guidance relating to standards development. Numerical benchmarks quantified desired environmental conditions, while methodological support and guidance consisted of tools to support the development of standards. The term provisional was used to qualify ideal performance standards of either format developed in situations in which the science was limited and additional basic and applied research would be required before final standards could be identified nationally or regionally (Roberts 2009). 

Scale of NAESIStandards

NAESIstandards were developed for a range of geographic scales, as Figure 4 shows. Regions smaller than 1 km2 in area were classified as farm or plot-level, regions 10–50 km2were designated as sub-watersheds, and regions with areas of hundreds or thousands of km2 were referred to as watersheds/ecoregions or ecozones, respectively. As illustrated, ammonia/particulate matter standards (developed under the air theme) applied to ecozones and watersheds/ecoregions. Standards developed under the water and pesticides themes were applicable at smaller scales (watershed/ecoregions and sub-watershed). Biodiversity standards were most versatile, being applicable to the broadest range of scales. The Overarching Report pointed out that, with the exception of some biodiversity standards, “NAESIstandards are not developed for application at the level of the individual farm.” Rather, “they have been designed with the intention for application within environmental boundaries (such as watersheds or ecoregions) to provide a landscape approach to environmental boundaries rather than political or land-ownership boundaries” (Bowerman et al. 2009).

Figure 4. Scale of NAESI standards 

Figure 4. Scale of NAESIstandards 

Source: Canada, 2007c, p. 3

NAESIThematic Areas

NAESIdeveloped standards in four thematic areas: water, biodiversity, pesticides and air. The goals for each theme were as follows:

  • Water: “Develop quantitative or qualitative environmental performance standards that provide benchmarks for nutrients, sediments and microbial pathogens in surface waters, as well as unsustainable water withdrawals for agricultural activities” (Bowerman et al. 2009, p. 20).
  • Biodiversity: “Deliver a suite of measurable standards representing levels of habitat quantity and quality necessary to support biodiversity conservation, applicable to key combinations of agricultural production types and landscapes across Canada” (Bowerman et al. 2009, p. 18).
  • Pesticides: Produce a suite of standards that may “inform and guide future efforts in the reduction of environmental risks posed by the use of pesticides in agriculture” (Bowerman et al. 2009, p. 19).
  • Air: “Produce a suite of regional air quality standards that would provide benchmarks against which the impacts of agricultural operations on Canadian air quality can be assessed” (Bowerman et al. 2009, p. 17).
Table 2: NAESI’s four thematic areas
Thematic areaKey priority areasSub-priorities
WaterNutrients, pathogens, instream flow needsSediments, water availability
BiodiversityHabitat conservationWetlands, riparian, connective corridors 
Pesticides20 priority pesticides in waterMeteorological, commodities, mixtures, pesticide risk curves
AirAmmonia as precursor to particulate matter 

Source: Roberts 2006; Bowerman et al. 2009, sec. 2.0

Key priority areas under the water theme were nutrients, pathogens and instream flow needs. For the biodiversity, pesticides and air thematic areas, key priorities were habitat conservation, priority pesticides in water, and ammonia as a precursor to particulate matter, respectively. Additional subjects were initially considered for inclusion as priorities but ended up not being part of NAESI’s final products. These subjects included water conservation, pesticides in the air, and odour, under the water, pesticides and air themes, respectively. The Overarching Report provided detailed explanations of the factors leading to the exclusion of these topics (Bowerman et al. 2009, sec. 4.0). 

Research undertaken through NAESIled to the development of 98 standards across the four thematic areas. Tables 6-1 through 6-4 in Appendix 6 list the number of standards for each thematic area, each according to whether the standard was classified as an achievable performance standard, a numerical descriptive benchmark, or a methodological or guidance-based standard.

Top of Page

3.0 Evaluation Design

3.1 Purpose and Scope of Evaluation

Since NAESIhas ended and is not being considered for renewal, a small-scale evaluation of limited scope was conducted, with a focus on identifying lessons learned rather than recommendations for improvement. Lessons learned may serve to inform the design of future initiatives in the agri-environmental sector.

The following issues were examined in this evaluation:

  • Relevance: Was the initiative consistent with departmental and government-wide priorities?
  • Design and delivery: Was the initiative designed and delivered in the best possible way?
  • Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?
  • Cost-effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?

A matrix of the evaluation questions, indicators and data sources that guided the evaluation appears in Appendix 1.

The evaluation focused primarily on the five-year timeframe of NAESI(2003–2004 to 2007–2008) but also encompassed some activities in 2008–2009, such as the final NAESIstakeholder workshop, which was held in January 2009. 

Top of Page

3.2 Evaluation Approach and Methodology

The methodology relied on two qualitative lines of evidence: interviews and document review.

Interviews with Stakeholders

A total of 18 interviews were completed with representatives from Environment Canada, Agriculture and Agri-Food Canada, other federal government departments (such as Health Canada’s Pest Management Regulatory Agency), provincial governments and other interested parties, such as environmental non-governmental organizations and industry. Of the 18 interviews, 10 were conducted with federal representatives, 3 with provincial representatives and 5 with representatives of environmental non-governmental organizations and industry. The interviews followed a guide that reflected the evaluation matrix (see Appendix 1). Separate guides were created for the interviews with representatives of the federal government, provincial governments, and environmental non-governmental organizations and industry (see Appendix 2). Interviewees received a copy of the guide prior to the interviews, which were conducted by telephone. Audiotaping, with the interviewee’s permission, ensured all important details were captured in the written notes.

Document Review

A review of relevant documents and literature provided background to prepare a profile of the initiative and respond to specific evaluation questions. The profile is a synopsis of NAESI, including program objectives, the roles and responsibilities of Environment Canada, Agriculture and Agri-Food Canada and other parties, and expected outputs and outcomes. Environment Canada provided most of the documents for review, including the Memorandum of Understanding, the Results-Based Management and Accountability Framework/Risk-Based Audit Framework for NAESI, the framework agreement for Growing Forward, the technical and synthesis reports (including the Overarching Report), and the proceedings of the 2006 and 2009 stakeholder and 2008 technical workshops. Additional documents, such as the Report on Plans and Priorities for each of the lead departments, the 2008 federal budget and the 2007 Speech from the Throne, were found online.

Top of Page

3.3 Limitations of the Evaluation 

The evaluation relied on qualitative evidence drawn from the 18 interviews and the document review. The number and type of stakeholders that could conceivably be affected by the standards and guidelines considerably exceeded the scope of the interviews conducted. Consequently, it is possible that some viewpoints may not have been captured or, more likely, certain perspectives were not emphasized sufficiently. At the same time, the diversity of interviewees was sufficient to offer some confidence that all essential perspectives were recorded. 

The evaluators did not assess the scientific validity of the standards and other technical documentation. The existence of considerable published information and the opinion of those not directly involved in their production support a conclusion that the standards do represent a technically valid system that can guide agri-environmental programming. Of course, the final validation of this will be whether NAESI outputs become the reference for future agri-environmental programming in Canada. Currently, it is too soon to tell.

The final limitation is that this evaluation did not assess the realization of long-term strategic outcomes because NAESI did not involve developing an implementation plan, and the degree to which the standards will be implemented is unclear.

Top of Page

3.4 Evaluation Ratings 

For each evaluation question, a rating was assigned to denote the level of progress made in achieving intended results and to give the reader with an overall view of the performance of NAESI. The ratings are based on a judgment of whether the findings indicate the following:

  • that the intended outcomes were achieved;
  • that some progress was made toward the intended outcomes; or
  • that little progress to date  was made toward the intended outcomes, and it is too early to see the full results.

In situations in which there was compelling subjective evidence that the initiative did well with respect to a given outcome but a complete assessment could not be done due to a lack of performance data, a “~” is placed in front of the rating.

The ratings for all evaluation questions are summarized in Appendix 8.

Top of Page

4.0 FINDINGS

4.1 Relevance

Overall Findings

Relevance: Was the initiative consistent with departmental and government-wide priorities?

The initiative was consistent with departmental and government-wide priorities. NAESI supported the Government of Canada’s priority to protect the environment and it aligned with Environment Canada’s and Agriculture and Agri-Food Canada’s strategic outcomes, particularly those relating to air, water and soil quality, as well as biodiversity.  

  • Although NAESI is consistent with Growing Forward, which replaced the Agricultural Policy Framework, it is not clear how NAESI will be integrated into it.

Consistency with Departmental and Government Priorities

Relevance
Indicator(s)
Rating
1. Was NAESI consistent with current Environment Canada and Agriculture and Agri-Food Canada priorities and with government-wide priorities?
  • Consistency of NAESI’s goals with current priorities of both departments
  • Alignment of NAESI with the Growing Forward
  • Consistency of NAESI’s goals with current priorities of the Government of Canada
Achieved

Alignment with Environment Canada Priorities

An assessment of the plans and priorities of the two lead departments provides insight into the extent to which NAESI aligned with these departments’ priorities. Environment Canada’s 2007 Report on Plans and Priorities describes the Department’s three strategic outcomes: the restoration, conservation and enhancement of Canada’s natural capital, risk reduction and contribution to the well-being of Canadians through weather and environmental predictions and services, and protection of Canadians and their environment from the effects of pollution and waste (Environment Canada 2007). NAESI also appears to have aligned with a number of the program areas associated with these strategic outcomes. For example, strategic outcome 1 (Canada’s natural capital is restored, conserved and enhanced) is linked to the following program areas:

  1. Biodiversity is conserved and protected.
  2. Water is clean, safe and secure.
  3. Canadians adopt approaches that ensure the sustainable use and management of natural capital and working landscapes (Environment Canada 2007, p. 18).

Two of the common goals for the environment agreed upon under the Agricultural Policy Framework and incorporated into the goals of NAESI were to “reduce risks and provide benefits to the health and supply of water” and to “ensure compatibility between biodiversity and wildlife” (Agriculture and Agri-Food Canada 2003, sec. 24.1.1, 24.4.4). These aligned well with the first of the two program areas under strategic outcome 1. Moreover, NAESI’s goal of “[evaluating] standards attainable by environmentally-beneficial agricultural production and management practices” would appear to have aligned with the adoption of approaches that ensure the sustainable use of natural capital, the third program area under strategic outcome 1 (Canada, 2003, sec. 1.0). Given the above, it appears NAESI aligned with Environment Canada’s plans and priorities.

Interviewees reported that NAESI aligned well with Environment Canada’s mandate to preserve, restore and enhance Canada’s natural capital. They also said it supported the Department’s focus on air and water quality, and climate change. While climate change was not a NAESI priority, interviewees noted that it was an overarching issue that was considered in all of the work under the initiative. 

Alignment with Agriculture and Agri-Food Canada Priorities

Agriculture and Agri-Food Canada’s 2007–2008 Report on Plans and Priorities outlines three strategic outcomes guiding the activities of the Department: security of the food system, health of the environment and innovation for growth. The second of these was relevant to NAESI and encompassed several priority areas, one of which involved “developing the knowledge to improve the environmental performance of the Canadian agricultural system” (Agriculture and Agri-Food Canada n.d., p. 46). The report states the following:

Acting on agricultural risks presumes a sound knowledge of the environment, and specifically a better understanding of the relationship between agriculture and the environment. It is essential to understand the processes and mechanisms by which agricultural inputs, such as nutrients and pesticides, affect resources, such as air, water, soil, and biodiversity, under different agri-environmental landscapes and management practices (Agriculture and Agri-Food Canada n.d., p. 46).

This priority area appears to have aligned with NAESI’s goal of increasing understanding of relationships between agriculture and the environment (see “2.2 Objectives and Expected Results,” above); therefore, the objectives of NAESI appear to have aligned, in part, with the plans and priorities of Agriculture and Agri-Food Canada.

Alignment with Growing Forward

Interviewees agreed that when it was established NAESI aligned with Agriculture and Agri-Food Canada’s priorities, given that it formed part of the Agricultural Policy Framework. However, this framework has been replaced with Growing Forward, which is a federal/provincial/territorial framework agreement to guide government action in the area of agricultural policy from 2008 to 2012. 

Presenters from Agriculture and Agri-Food Canada at the 2009 Final Stakeholder Workshop (Carona Designs Inc. 2009) noted that NAESI standards could inform policy making with specific reference to the four elements of the Growing Forward policy suite (discover, develop, deliver and determine). This drew attention to the link between NAESI and the Growing Forwardpolicy outcome of developing an agricultural sector that contributes to society’s priorities, including “the capacity to meet the need for strong environmental stewardship and to deliver ecological goods and services” (Agriculture and Agri-Food Canada 2007, sec. 5.2). The evidence suggests that NAESI is consistent with Growing Forward. 

Nonetheless, many interviewees were uncertain whether and how NAESI would be integrated into this policy framework. While agreeing that Growing Forward continues to include the environment as a priority, interviewees said that increased emphasis will be placed on economic sustainability and profitability. They also noted that climate change and water are Growing Forward’s main priorities, with a few interviewees suggesting that biodiversity has become less of a priority.

Alignment with Government-Wide Priorities

Finally, in the 2007 Speech from the Throne, the Governor General identified the protection of the environment as a priority for the Government of Canada. However, the speech did not refer to NAESI or the environmental impact of agriculture, focussing instead on related concerns, such as greenhouse gas emissions and the consequent impact on climate change, as well as air pollution, environmental legislation and enforcement.

Top of Page

4.2 Design and Delivery

Overall Findings

Design and delivery: Was the initiative designed and delivered in the best possible way?

Overall, NAESI was well designed and utilized effective delivery mechanisms. Its strengths included having a well-defined Memorandum of Understanding and a designated program manager, which facilitated an effective partnership between the lead departments. NAESI focused on an appropriate set of thematic areas: water, biodiversity, pesticides and air. However, NAESI lacked strategies for integrating themes and implementing standards.

  • Some stakeholders and beneficiaries would have liked to have seen soils included as a theme; however, it was determined that Agriculture and Agri-Food Canada had the knowledge and expertise required to complete this work internally.
  • The partnership between the lead departments was integral to the development of the agri-environmental standards. However, some stakeholders and beneficiaries said the partnership should have been better balanced.
  • Some stakeholders suggested that the initiative should have involved a wider range of partners and collaborators.
  • The governance structure was effective, and the Memorandum of Understanding clearly defined the roles and responsibilities of the lead departments. There was good communication between the departments.
  • NAESI was completed on time and budget; however, it was not able to deliver on its intended outcomes during the limited time available. Definitive standards could not be developed for all of the sub-themes, and there was limited opportunity to integrate the themes.

Design

Design and delivery
Indicator(s)
Rating
2. Was the design of NAESI sound?
  • Fact-based observations on the appropriateness of aspects of the NAESI design: the four themes selected, the degree of integration among the four themes, and the focus on both ideal and achievable standards 
  • Views on strengths and best practices, weaknesses and needed improvements to the NAESI design
Some progress

As noted, the formal purpose of NAESI was to develop science-based agri-environmental standards, not including formulating a plan to implement the standards; however, several interviewees said that the intent of NAESI was unclear and that more forethought about how the standards would be used was required (see “4.3 Success,” below). These comments indicate that NAESI did not include an explicit implementation or adoption plan in its design or resource allocation.

Many interviewees also said that the terminology used to describe the outputs created ambiguity about how outcomes would be realized. They explained that the name of the initiative and the terminology in the Memorandum of Understanding suggested that NAESI was intended to lead to the development of agri-environmental standards and that the standards would, in turn, be used to establish regulations for the agricultural sector. At the same time, however, interviewees said that they understood that the intent of NAESI was not to establish standards for regulatory purposes but rather to establish non-regulatory standards, guidelines or performance targets. Several interviewees noted that the softer language used to describe aspects of the initiative (e.g. the word guidelines) may have compromised some stakeholders’ support for the initiative. 

Most interviewees said that the themes selected for inclusion in NAESI were appropriate. They explained that the lead departments worked collaboratively to identify and select a set of themes that would meet their respective needs. Nonetheless, a few interviewees questioned why pesticides was chosen as a theme, noting that pesticides are a stressor that can cause an adverse impact on the environment, in contrast to water, biodiversity and air, which focus on receptors that respond to environmental stressors. Some interviewees suggested that soils should have been included as a theme. However, they noted that Agriculture and Agri-Food Canada had the internal knowledge and expertise required to pursue this topic independently; a few interviewees reported that they had not seen any evidence this department had undertaken this work. Other suggestions for modifications to the themes were to include biodiversity and climate change as overarching themes and to integrate pesticides within the other three themes.

Many interviewees said that a key feature of NAESI’s design was the partnership between the two lead departments. The interviewees reported that the collaboration between the departments was integral to the development of agri-environmental standards, since each brought specific expertise to the table. In addition to fostering inter-departmental relationships, NAESI also created the opportunity for scientists from various sections of Environment Canada to work together. 

Some interviewees suggested that NAESI should have involved a wider range of partners and collaborators, such as provincial governments, and industry and producer associations. However, respondents representing industry associations noted that it is difficult for them to know when to get involved in government programs. Some of the industry associations said that they lack the technical knowledge required to contribute to an initiative such as NAESI. For example, the representative of one producer association said, “We rely on [Agriculture and Agri-Food Canada] to tell us if it is good science.”

Implementation Success

Design and delivery
Indicator(s)
Rating
3. Was NAESI successfully implemented?
  • Knowledge of the effectiveness of the implementation of NAESI (i.e. the overall initiative, not the standards)
  • Views on strengths and best practices, weaknesses and needed improvements to the implementation approach for NAESI
Some progress

Interviewees who could comment on the implementation of NAESI indicated that it was delivered on time and within budget. However, several noted that the time frame within which the work was to be completed limited the capacity of the initiative to deliver on its strategic outcomes. 

Several interviewees said that the first year of the initiative was spent identifying the themes and setting priorities. Some interviewees suggested that this process could have been completed more quickly. They also noted that this planning phase effectively reduced the amount of time available for the research, standards development and reporting.

While there was the intent to integrate some of the themes, interviewees said that there was limited opportunity to accomplish this. One reason NAESI was unable to thoroughly integrate the themes was that the majority of the time available was spent conducting scientific research and developing the standards. Interviewees noted that each of the standards had to be developed independently prior to considering how they could be integrated. Nonetheless, efforts were made by those working on each theme to keep those working on other themes apprised of their direction and progress. Another reason the themes were not well integrated was that the results of the scientific research conducted were required to inform decisions about how the themes could be integrated. Interviewees reported that discussions about how to integrate the themes are underway; however, they said that without additional funding, this integration might not occur. 

Some interviewees mentioned that integration of the themes is critical to understanding how the standards interact and what potential exists for adverse effects within or across themes. Some interviewees said that preliminary attempts to integrate some of the themes (pesticides and water, and water and biodiversity) were made.

Despite these challenges, interviewees said NAESI resulted in good science and led to the development of some standards. Many noted that it was simply not possible for standards to be developed for each theme and priority within the time available. 

Governance and Communication Processes

Design and delivery
Indicator(s)
Rating
4.  How effective was the governance structure of NAESI?
  • Knowledge and documentation on the clarity of the respective roles and responsibilities of the two lead departments
  • Knowledge and documentation on the effectiveness of collaboration and communications between the two lead departments
  • Knowledge of the effectiveness of the partnership and the leadership role of each department on NAESI projects
Achieved

Most interviewees agreed that the governance structure for the initiative was effective. The Memorandum of Understanding between the two lead departments and the Treasury Board of Canada Secretariat clearly defined the roles and responsibilities of each department. Environment Canada was responsible for leading each theme, including developing the required work plans, budgets and reports, and conducting the majority of the science. Agriculture and Agri-Food Canada had an advisory role and worked with Environment Canada to identify the thematic areas, provided suggestions for the direction of projects, and reviewed deliverables and reports. 

Some interviewees suggested that there should have been more of an equal partnership between the departments, with them co-leading each of the themes. However, others thought this might be problematic given the departments’ differing mandates and priorities. A few interviewees noted that the theme leads should not have been asked to act as managers for the initiative, since conflicts of interest could arise given the leads’ vested interest in the research. Some interviewees said that inequalities in the partnership negatively influenced departmental commitment to the initiative; some interviewees mentioned that they had observed tension between representatives of the two departments. 

Interviewees reported that one of NAESI’s strengths was the designated program manager, who served as a single point of contact. They noted that the program manager was integral to ensuring effective communication between the lead departments, which occurred, for the most part. 

Top of Page

4.3 Success

Overall Findings

Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?

The initiative was effective in achieving the immediate and intermediate outcomes, and the ultimate outcomes outlined in the logic model. NAESI resulted in the development of 98 standards across four theme areas, and the information generated through the initiative was effectively communicated to stakeholders. However, many of the standards have not been implemented, and a strategy to facilitate their implementation has not been developed.

  • NAESI contributed to the federal government’s understanding of the relationships and links between agriculture and the environment. It expanded Agriculture and Agri-Food Canada’s knowledge base and increased Environment Canada’s understanding of agricultural production practices.
  • NAESI generated the scientific research needed to develop standards and lay the foundation for the creation of other standards. The standards can serve as benchmarks for environmental performance and be used to inform decision making.
  • The extent to which NAESI resulted in the development of standards varied across themes. The largest number was developed under the pesticides theme. More work is needed in the areas of water availability, instream flow needs and biodiversity, the standards which are currently designated as provisional. At this time, scientific understanding is inadequate to recommend an approach or a numerical or descriptive benchmark; further analysis and research must be undertaken before it is possible to identify standards. Some standards are national in scope and others are location-specific.
  • At this point, the standards appear not to figure in the programming or policies under Growing Forward, which is the key mechanism for implementation.

Immediate and Intermediate Outcomes

Success
Indicator(s)
Rating

5. To what extent did NAESI achieve its intended immediate and intermediate outcomes?

a) Increased understanding of the relationships and links between agriculture and the environment through the collection of scientific information

  • Change in level of awareness and understanding of the links between agriculture and the environment by all stakeholders (Environment Canada, Agriculture and Agri-Food Canada and industry)
  • Incorporation of science results into standard development
~Achieved

Note: The “~” symbol refers to the fact that although there was compelling subjective evidence that the initiative did well with respect to this evaluation question, a complete assessment could not be done due to a lack of performance data.

Interviewees reported that one of NAESI’s strengths was its contribution to the federal government’s understanding of the relationships and links between agriculture and the environment. Interviewees said that although Agriculture and Agri-Food Canada already had a strong awareness of the connections between agriculture and the environment, NAESI expanded upon this knowledge base. It also increased this department’s awareness of Environment Canada’s expertise. In addition, NAESI raised Environment Canada’s awareness of agricultural production practices, particularly in the context of working landscapes and business decisions. It also helped Environment Canada understand some of the agri-environmental issues that its counterpart department and the agricultural sector face daily. 

Some interviewees emphasized that NAESI did not increase awareness of issues but rather added to the knowledge of the interactions between agriculture and the environment. The interviewees explained that prior awareness of the issues that needed to be addressed contributed to the identification of the themes that were included in NAESI.

Success
Indicator(s)
Rating

5. To what extent did NAESI achieve its intended immediate and intermediate outcomes?

b) Effective technology transfer packages and communications support materials for a range of audiences (e.g. technical, policy-makers and stakeholders)

  • Number of technology transfer packages and presentations delivered
  • Number and types of stakeholder groups receiving communications on NAESI
  • Level of awareness and understanding of NAESI results and standards among stakeholders
Achieved

The evaluation findings indicate that this outcome was achieved. According to data provided by the NAESI Secretariat at Environment Canada, 17 technology transfer packages were delivered in various forms, including five National Agri-Environmental Coordinating Committee meetings, two stakeholder meetings, four annual technical series CDs and one final synthesis CD, one brochure and four newsletters. In addition, approximately 75 technical presentations were delivered over the life of the program to a range of stakeholders (e.g. provinces, researchers, academia, conservation authorities, industry and participants at national and international conferences).

Appendix 4 sets out NAESI’s Communications Strategy. A communications specialist ensured that NAESI materials and reports were clear and consistent in tone. The breadth and depth of NAESI communication materials contributed to the transparency of the initiative. Most interviewees said that NAESI did a good job of communicating with stakeholders, but some suggested that NAESI could have engaged stakeholders earlier in the process.

Success
Indicator(s)
Rating

5. To what extent did NAESI achieve its intended immediate and intermediate outcomes?

c) Development of methods for ideal and achievable performance standards

  • Number of methods developed
  • Knowledge of the effectiveness of methods developed
Achieved

Data from the NAESI Secretariat indicate that 36 methods for standards were developed. These include 21 methods developed as of March 2007 and 15 methods for the final suite of standards (one for the air theme, two for biodiversity, six for pesticides and six for water).

Interviewees noted that solid scientific research was conducted through NAESI. Some of the work expanded on existing information and other work was new research. Interviewees said NAESI generated the scientific research needed to develop standards and lay the foundation for the development of other standards. 

Interviewees explained that ideal performance standards represent the best state of the world, while achievable performance standards represent what can be achieved given the current state of the environment. They noted that the development of the standards was science-driven and that standards simply could not be developed without a scientific foundation. Interviewees reported one of the strengths of NAESI was that all of the research was peer-reviewed.

Ultimate Outcomes

Success
Indicator(s)
Rating

6. To what extent did NAESI progress toward its intended ultimate outcomes?

  • Standards (ideal and achievable) for priorities--air (particulate matter/ammonia and odour); biodiversity (habitat conservation); pesticides (high-risk, commodity); and water (nutrients, pathogens, sediment, instream flow needs, quantity and conservation)--that are national, transparent and coherent with existing roles and responsibilities (regulations) across government and non-government sectors
  • Number and types of standards developed versus identified priorities
  • Of the priority pesticides identified, percentage of pesticides for which standards were developed
  • Number of semi-annual status updates and progress reports for air theme
  • Number of semi-annual status updates and progress reports for biodiversity theme
  • Number of semi-annual status updates and progress reports for pesticide theme
  • Number of semi-annual status updates and progress reports for water theme
  • Knowledge and documentation on the extent to which the standards are national, transparent and coherent with existing roles and responsibilities across the government and non-government sectors
Achieved

Appendix 5 summarizes the contents of four NAESI newsletters issued between 2006 and 2008, which offered a detailed account of NAESI’s progress in developing the standards. As discussed earlier, 98 standards were developed across the four thematic areas. Table 3 summarizes these standards, based on the information in Appendix 6.

Table 3: Number of NAESI standards by thematic area and class
Thematic areaAchievable performance standardsIdeal performance standardsTotal
Numerical or descriptive benchmarkMethodological or Guidance
Water3(1 provisional) 11 (2 provisional)16
Biodiversity1717
Pesticides12411164
Air11
Total16(1 provisional) 69(2 provisional) 1398

 

Source: Bowerman et al. 2009, sec. 3.0; Roberts 2009

As set out in Appendix 6, the water, biodiversity, pesticides and air themes produced 16, 17, 64, and 1 standard, respectively. NAESI Secretariat data indicate that the 64 standards developed in the pesticides thematic area represent 100 percent coverage of the 20 priority pesticides identified. In the process of developing the standards, seven semi-annual status updates were provided for the four themes (in March and November 2005, March and November 2006, March and November 2007, and March 2008).

Of the 98 standards, 69 were numerical or descriptive benchmarks, while 13 were methodological or guidance-based. These two types of standards are both categorized as ideal performance standards and account for 82 of the 98 standards. The remaining 16 standards were achievable performance standards (see Tables 6-1 to 6-4 in Appendix 6.) Based on this information, it is apparent that NAESI developed standards in the required themes and sub-themes within the terms of the Memorandum of Understanding.

Interviewees reported that the extent to which NAESI developed standards varied across themes. They explained that this was because the initiative’s ability to establish standards was directly related to the maturity and completeness of the information available upon which standards could be based. In areas in which there was a solid foundation of existing information and knowledge, NAESI was able to articulate standards. When there were gaps in information, NAESI was only able to conduct some of the science needed to set the standards. In these cases, additional research is needed before the standards can be fully defined. Interviewees said that the initiative was the most successful in developing standards under the pesticides theme, with the most work remaining to be done in the areas of water availability, instream flow needs and biodiversity. Interviewees also noted that although odour was originally included under the air theme, it was removed because the scientific foundation was underdeveloped.

Usefulness of Standards

Success
Indicator(s)
Rating
7. To what extent are the NAESI standards useful?
  • Knowledge of the potential uses and usefulness of NAESI standards and the degree to which they can be implemented in each of the four themes from the perspective of the lead departments and other stakeholders
  • Whether programs are in place to implement NAESI standards
  • Timelines for application of NAESI standards
  • Knowledge of the extent to which NAESI has facilitated synergies across program areas of the environment chapter of the Agricultural Policy Framework
  • Knowledge of the extent to which Environment Canada has leveraged results from NAESI that are useful for other activities, such as other regulatory processes
Little progress to date*

*Although early progress was made in identifying potential uses for some of the NAESI standards, it is not clear whether and how many of the standards will be implemented.

Some interviewees described NAESI as a science program that had the potential to lead to standards development in some areas. They indicated that the timeframe for the initiative was too short to accommodate the completion of the scientific research as well as the implementation of the standards. 

Interviewees had varying opinions on how the standards could be implemented and said that the standards are at various stages of readiness for implementation. Most interviewees said that the standards under the themes of pesticides and water had the most potential for implementation. For example, some of the pesticide standards have been circulated to the Canadian Council of Ministers of the Environment for consideration as Canadian Water Quality Guidelines.

A few interviewees stated that the standards were initially being developed for use in farm certification. However, that concept was abandoned and no clear options for how else the standards could be used were developed. Nonetheless, those involved with NAESI proceeded under the impression that Agriculture and Agri-Food Canada would use the results in its programs. Interviewees said that they do not have a clear understanding of this department’s intentions in this regard and said that a solid implementation plan was not established at the onset of the initiative.

Some interviewees were uncertain how the information and standards generated through NAESI could be utilized, indicating that the work completed through this program is premature and that additional research is needed before the standards will be useful. These interviewees fear that if NAESI were not continued or built upon, the expenditures made through this initiative would be lost.

Other interviewees noted that the standards serve as benchmarks for environmental performance. They noted that the standards can be used to measure the state of the environment and assess the effectiveness of agri-environmental programs. Some interviewees mentioned that provinces are considering how the standards can be used to develop performance standards and inform their work in areas such as land-use planning, environmental farm planning, and the development of ecological goods and services. 

Several interviewees would like to see NAESI incorporated into the development of on-farm programs to demonstrate the impact of agriculture on environmental degradation and to establish beneficial management practices. The interviewees said that the standards could be used as guidance for environmental mitigation and to identify target areas for environmental improvement. 

Some interviewees said the standards are best suited to serve as tools to inform decision making. In some cases, the standards are national, while others are location-specific. This means that users may need to conduct additional research before they can apply the standards to their situation. 

Stakeholders and beneficiaries did not share a common understanding of how the standards developed through NAESI would be used. For some, the name of the initiative and the terminology used to describe it contributed to this lack of clarity. For them, the stated goals seem to suggest that NAESI would result in the development of standards that would be used to establish regulations for the agricultural sector; however, other stakeholders thought that NAESI would develop non-regulatory standards, guidelines or performance targets. The fact that developing implementation processes was not part of the initiative may have contributed to this ambiguity.

Top of Page

4.4 Cost-Effectiveness

Overall Findings

Cost-effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?

While a cost-effective approach was used to implement NAESI, it is not clear whether the initiative was good value for money. NAESI resulted in good science, but unless the standards it developed are implemented, it will not realize its long-term strategic outcomes, which will limit the value of the initiative for the federal dollars invested.  

  • The administration costs for NAESI were minimal, and the initiative partnered with other projects when possible.
  • NAESI did not duplicate other programs. 
  • NAESI had links with programs such as the National Agri-Environmental Health Analysis and Reporting Program, Watershed Evaluation of Beneficial Management Practices, Building Public Confidence Initiative, and organizations such as the Canadian Council of Ministers of the Environment. 
  • Stakeholders and beneficiaries said that they thought that NAESI was a reasonable approach and had few suggestions for alternative approaches. Some said NAESI should not have taken a policy approach. Others said that it will be important to consider the social factors and economics associated with the standards.

Interviewees said that it was difficult to judge whether NAESI was good value for money. They indicated that a substantial amount of funding was allocated to this initiative but that it was not clear what role NAESI would play in the realization of the long-term strategic outcomes. Although NAESI resulted in good science and produced numerous outputs, more work is needed before some of the information can be used. In addition, not all of the information generated is national in scope.

Other interviewees said that NAESI was good value for money because it provided standards, generated knowledge and science for other areas, and created a partnership between the two lead departments. Interviewees mentioned that the majority of the funding was used for research; admnistration costs were minimal. NAESI also made attempts to partner with other projects, when possible. For example, there were partnerships with the Pesticide Science Fund, the Watershed Evaluation of Beneficial Management Practices initiative, and regional air monitoring stations. 

Several interviewees indicated that some of the themes were not advanced enough given the amount of funding they were allocated. Nonetheless, there were some areas in which the initiative recognized that it would not be able to make a meaningful contribution. In these cases, the work was dropped and another direction pursued. Some interviewees suggested that the government’s three- to five-year funding cycles are too short for science programs to conduct research and demonstrate impacts. 

Interviewees also said that those involved in the sub-themes worked together closely when possible and took advantage of local resources. For example, under the water theme, scientists worked together on field monitoring and hired local farmers and university students to conduct some of the fieldwork so they could minimize travel costs. Under some themes, staff members were able to leverage money from other federal departments. For example, those involved in the air theme engaged in a data-gathering case with Statistics Canada.

Links to Existing Programs

Cost-Effectiveness
Indicator(s)
Rating
8. To what extent did NAESI duplicate, overlap with or complement other existing government programs?
  • Extent to which those involved with the initiative are aware of duplication, overlap or complementarity of NAESI with other federal, provincial or territorial programs
  • Knowledge and documentation on ways in which NAESI is or is not unique
Achieved

Interviewees were not aware of other programs that are developing agri-environmental standards. Nonetheless, NAESI had links to some existing programs, such as the National Agri-Environmental Health Analysis and Reporting Program (NAHARP) and the Watershed Evaluation of Beneficial Management Practices (WEBs). 

Interviewees explained that NAHARP is developing a suite of standards that could be used to assess the current state of the environment, against which future programs for environmental sustainability can be benchmarked. Interviewees noted the design of NAHARP and NAESI are similar. Like NAESI before it, NAHARP is utilizing a collaborative approach, addressing a range of issues and involving numerous scientists. In addition, both programs are science-based. 

WEBs is measuring the economic and water quality impacts of selected beneficial management practices. Some interviewees noted that some of the projects proposed for NAESI were too closely related to WEBs and, therefore, not funded. However, there was some overlap related to work that explored how implementing a beneficial management practice would impact various standards.

Other programs that NAESI had links to are the Building Public Confidence in Pesticide Regulation and Improving Access to Pest Management Initiative of the Pest Management Regulatory Agency, the Canadian Council of Ministers of the Environment’s Canadian Environmental Quality Guidelines and Water Quality Guidelines for the Protection of Aquatic Life. 

Alternative Approaches

Cost-Effectiveness
Indicator(s)
Rating
9. Were there alternative, more cost-effective ways of achieving the intended outcomes of NAESI?
  • Knowledge of the extent to which NAESI has provided good value for money to date
  • Suggestions for improving the efficiency and cost-effectiveness of the initiative
Achieved

Some interviewees found it difficult to comment on whether NAESI was the best approach to developing agri-environmental standards because there are no clear impacts of the standards to date and it is uncertain how the standards will be used in the future. However, many noted that NAESI was a reasonable approach, and only a few had suggestions for alternative approaches. Some interviewees suggested that NAESI was a science program and, therefore, it should not have been expected to lead to the implementation of the standards it developed. A few interviewees said that it would be important to consider the social factors and economics associated with the standards. 

Top of Page

5.0 Conclusions

NAESI featured collaboration between Environment Canada and Agriculture and Agri-Food Canada on the development of agri-environmental standards. The partnership and relationship created between these departments is seen as one of the main strengths of NAESI. By working together on this initiative, the departments gained an awareness and understanding of the knowledge and expertise of each. 

Although NAESI successfully contributed to the scientific knowledge base of the interactions between agriculture and the environment, it was not able to develop a complete set of national standards for each theme, simply because the science is not sufficiently advanced to support a quantitative benchmark. NAESI did not explicitly include implementation processes or resources, which has created uncertainty about the realization of the long-term strategic outcomes.

Top of Page

5.1 Main Findings

A key finding emerging from this evaluation is the concern of all interviewees about how the standards developed will contribute to the initiative’s expected strategic outcomes. NAESI was a science program designed to establish standards; however, for many stakeholders it remains unclear how the standards will be implemented and utilized. While there were several suggestions for their role--for example, to serve as a baseline for tracking environmental performance, to inform decision making and to form a foundation for future research--interviewees indicated that an implementation strategy for the initiative had not been defined. For example, NAESI appears not to have found expression in Growing Forward, the successor to the Agricultural Policy Framework.

Relevance

The evaluation finds that NAESI aligned with Environment Canada’s strategic outcomes and program areas. It also was part of the Environment Pillar of Agriculture and Agri-Food Canada’s Agricultural Policy Framework and logically is also consistent with Growing Forward, that framework’s successor. At this time, however, it is unclear how NAESI will be integrated into Growing Forward.

Design and Delivery

Stakeholders all said that the four thematic areas were appropriate for agri-environmental programming. The partnership between the two lead departments was also seen as reflecting good design practice, as was the practice of including stakeholders such as industry groups. A few stakeholders said that wider consultation would have been beneficial. Support for the governance structure and the appointment of a single designated program manager came from all interviewees, who saw these as effective delivery processes.  

Success

NAESI increased scientific understanding of the relationship between agriculture and the environment and developed a total of 98 standards. While some variation exists in the number and completeness of the standards developed across the four thematic areas, and work remains to be completed in some areas, interviewees said that a significant body of research has been synthesized to support agri-environmental policy and programming.

For some interviewees the fact that NAESI was intended to support the development of non-regulatory standards, guidelines and performance targets created ambiguity, since the strategic outcomes clearly require implementation and adoption within programs and regulations. Without an implementation plan, however, it is unclear how the NAESI standards will result in the long-term strategic outcomes.

Cost-Effectiveness

The administrative costs of NAESI were low, and the initiative did not duplicate existing programming in Canada. NAESI linked to important federal agri-environmental initiatives, and most interviewees said that the initiative generated state-of-the art scientific information; however, it is difficult to judge the value of NAESI, since it is unclear whether and how the standards developed will be implemented.

Top of Page

5.2 Key Lessons Learned 

Since NAESI has ended, the focus of this evaluation was on lessons learned as opposed to recommendations for improvement. Lessons learned from the experience with NAESI may serve to inform the design of future similar initiatives.

The following are the lessons learned from NAESI.

  1. Clearly stating the intent and objectives of the initiative at its outset, especially with respect to implementation, increases partner and stakeholder support and buy-in. This also involves ensuring all stakeholders understand the language used in memoranda of understanding and communication materials. Had this happened for this initiative, work on the incomplete standards may have increased and stakeholder implementation of NAESI’s outputs may have been encouraged.
  2. A single program manager and point of contact proved instrumental to this horizontal initiative. The manager facilitated interdepartmental communication and coordinated diverse technical initiatives.
  3. Maintaining strong relationships and communication with partners is essential not only to the creation of an initiative’s intended outputs but also for the realization of its intended outcomes. For an initiative such as NAESI, the standards developed must be integrated into agri-environmental programming that supports on-farm adoption of beneficial management practices. Agriculture and Agri-Food Canada has primary responsibility for ensuring that NAESI outputs become integrated into the agri-environmental programming under Growing Forward. 
  4. The primary objective of NAESI was to develop agri-environmental standards. While NAESI developed national standards, synchronization with existing or planned provincial standards or regulations remains unfinished. Consultation with provincial stakeholders at an earlier stage of the initiative might have enabled NAESI to become integrated with provincial processes. This might also have helped stimulate additional support for the initiative. 
  5. Defining an implementation strategy for the outputs is essential for their integration into agri-environmental programming. Although NAESI added to the scientific knowledge base and established standards, it lacked an implementation strategy; therefore, it cannot be said that it has progressed toward its strategic outcomes. In the final analysis, realization of the strategic outcomes will require an implementation plan to promote the use and application of the standards through the agri-environmental programming under Growing Forward.

Top of Page

Appendix 1

Evaluation Issues and Questions

Evaluation Matrix
QuestionIndicatorsData sources and methods
Relevance: Was the initiative consistent with departmental and government-wide priorities?
1. Was NAESI consistent with current Environment Canada (EC) and Agriculture and Agri-Food Canada (AAFC) priorities and with government-wide priorities?
  • Consistency of NAESI’s goals with current priorities of EC and AAFC
  • Alignment of NAESI with Growing Forward
  • Consistency of NAESI’s goals with current priorities of the Government of Canada

Documents and data reviewed

  • EC and AAFC Reports on Plans and Priorities (2007–2008)
  • Growing Forward (2007)
  • Speech from the Throne (October 2007)
  • Federal Budget (February 2008)

Interviewees

  • EC
  • AAFC
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
Design and delivery: Was the initiative designed and delivered in the best possible way?
2. Was the design of NAESI sound?
  • Fact-based observations on the appropriateness of aspects of the NAESI design--the four themes selected, the degree of integration among the four themes, and the focus on both ideal and achievable standards 
  • Views on strengths, best practices, weaknesses and needed improvements to the NAESI design

Documents and data reviewed

  • Results-Based Management and Accountability Framework/Risk-Based Audit Framework
  • Memorandum of Understanding
  • NAESI Technical Series (2005–2008)
  • NAESI Stakeholder Consultation Workshop proceedings (March 2006)

Interviewees

  • EC
  • AAFC
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
3. Was NAESI successfully implemented?
  • Knowledge of the effectiveness of the implementation of NAESI (i.e. the overall initiative, not the standards)
  • Views on strengths and best practices, weaknesses and needed improvements to the implementation approach for NAESI

Document and Data Review

  • Results-Based Management and Accountability Framework/Risk-Based Audit Framework
  • Memorandum of Understanding
  • NAESI Technical Series (2005–2008)
  • Minutes of interdepartmental meetings
  • NAESI Stakeholder Consultation Workshop proceedings (March 2006)

Interviewees

  • EC
  • AAFC
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
  • Other stakeholders (e.g. environmental non-governmental organizations, industry)
4. How effective was the governance structure of NAESI?
  • Knowledge and documentation on the clarity of the respective roles and responsibilities of EC and AAFC
  • Knowledge and documentation on the effectiveness of collaboration and communications between EC and AAFC
  • Knowledge of the effectiveness of the EC–AAFC partnership and the leadership role of each department on NAESI projects

Documents and data reviewed

  • Results-Based Management and Accountability Framework/Risk-Based Audit Framework
  • Memorandum of Understanding
  • NAESI Technical Series (2005–2008)
  • NAESI Stakeholder Consultation Workshop proceedings (March 2006) (Ellis 2006)
  • Minutes of interdepartmental meetings

Interviewees

  • EC
  • AAFC
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?

5. To what extent did NAESI achieve its intended immediate and intermediate outcomes?

a) Increased understanding of the relationships and links between agriculture and the environment through the collection of scientific information

 

  • Change in level of awareness and  understanding of the links between agriculture and the environment by all stakeholders (lead departments and industry)
  • Incorporation of science results into standard development

Documents and data reviewed

  • NAESI Technical Series (2005–2008)
  • NAESI newsletters
  • NAESI Stakeholder Consultation Workshop proceedings (March 2006) (Ellis 2006)

Interviewees

  • Environment Canada
  • Agriculture and Agri-Food Canada
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
  • Other stakeholders (e.g. environmental non-governmental organizations, industry)
b) Effective technology transfer packages and communications support materials for a range of audiences (e.g. technical, policy-makers and stakeholders)
  • Number of technology transfer packages and presentations delivered
  • Number and types of stakeholder groups receiving communications on NAESI
  • Level of awareness and understanding of NAESI results and standards among stakeholders
  • Same as above
c) Development of methods for ideal and achievable performance standards
  • Number of methods developed
  • Knowledge of the effectiveness of methods developed
  • Same as above

6. To what extent did NAESI progress toward its intended ultimate outcomes?

Standards (ideal and achievable) for priorities--air (particulate matter/ammonia and odour); biodiversity (habitat conservation); pesticides (high-risk, commodity); and water (nutrients, pathogens, sediment, instream flow needs, quantity and conservation)--which are national, transparent and coherent with existing roles and responsibilities (regulations) across government and non-government sectors

  • Number and types of standards developed versus identified priorities
  • Of the priority pesticides identified, percentage of pesticides for which standards have been developed
  • Number of semi-annual status updates and progress reports for the air theme
  • Number of semi-annual status updates and progress reports for the biodiversity theme
  • Number of semi-annual status updates and progress reports for the pesticide theme
  • Number of semi-annual status updates and progress reports for the water theme
  • Knowledge and documentation on the extent to which the standards are national, transparent and coherent with existing roles and responsibilities across the government and non-government sectors

Documents and data reviewed

  • NAESI Technical Series (2005–2008)
  • NAESI Draft Synthesis reports (January 2008)
  • Proceedings of Environment Canada/Agriculture and Agri-Food Canada Technical Meeting, Calgary (February 2008) (Belanger 2008)

Interviewees

  • Environment Canada
  • Agriculture and Agri-Food Canada
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
  • Other stakeholders (e.g. environmental non-governmental organizations, industry)
7. To what extent are the NAESI standards useful?
  • Knowledge of the potential uses and usefulness of NAESI standards and the degree to which they can be implemented in each of the four themes from the perspective of Environment Canada, Agriculture and Agri-Food Canada and other stakeholders
  • Whether programs are in place to implement NAESI standards
  • Timelines for application of NAESI standards
  • Knowledge of the extent to which NAESI has facilitated synergies across program areas of the environment chapter of the Agricultural Policy Framework
  • Knowledge of the extent to which Environment Canada leveraged results from NAESI that are useful for other activities (e.g. other regulatory processes) 

Interviewees

  • Environment Canada
  • Agriculture and Agri-Food Canada
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
  • Other stakeholders (e.g. environmental non-governmental organizations, industry)
Cost-Effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?
8. To what extent did NAESI duplicate, overlap with or complement other existing government programs?
  • Extent to which those involved with the initiative are aware of duplication, overlap or complementarity of NAESI with other federal, provincial or territorial government programs
  • Knowledge and documentation on ways in which NAESI is or is not unique

Documents and data reviewed

  • NAESI Technical Series (2005–2008)
  • NAESI newsletters
  • NAESI Stakeholder Consultation Workshop proceedings (March 2006) (Ellis 2006)

Interviewees

  • Environment Canada
  • Agriculture and Agri-Food Canada
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
  • Other stakeholders (e.g. environmental non-governmental organizations, industry)
9. Were there alternative, more cost-effective ways of achieving the intended outcomes of NAESI?
  • Knowledge of the extent to which NAESI has provided good value for money to date
  • Suggestions for improving the efficiency and cost-effectiveness of the initiative

Documents and data reviewed

  • Budgetary and financial data for NAESI
  • NAESI Technical Series (2005–2008)

Interviewees

  • Environment Canada
  • Agriculture and Agri-Food Canada
  • Other government departments (e.g. Health Canada, Pest Management Regulatory Agency)
  • Provincial and territorial governments
  • Other stakeholders (e.g. environmental non-governmental organizations, industry)

Top of Page

Appendix 2

Interview Guides

EVALUATION OF THE NATIONAL AGRI-ENVIRONMENTAL STANDARDS INITIATIVE

INTERVIEW GUIDE
FEDERAL GOVERNMENT REPRESENTATIVES

Introduction

The Audit and Evaluation Branch of Environment Canada (EC) is conducting an evaluation of the National Agri-Environmental Standards Initiative (NAESI). A brief overview of the program is appended to this guide. The objective of this evaluation is to assess issues related to the program’s relevance, design and delivery, success, and cost-effectiveness. 

EC has hired PRA Inc., a private research firm, to conduct the evaluation, which comprises a document review and key informant interviews. The questions below serve to guide the interview process. If you are unable to answer a specific question or it does not apply to you, please tell the interviewer and we will skip that question. Please note that your responses will not be linked to you in the evaluation report.

1. Please briefly describe your relationship to and/or involvement with NAESI.

  • For how long have you had some involvement with the initiative?
  • With which themes (air, biodiversity, pesticides, and water)/standards have you been most involved? What were your specific roles and responsibilities?
  • What stakeholders (federal or provincial governments, producer organizations, non-profit organizations, etc.) have you worked with as part of this initiative?

Relevance

2. How well does NAESI align with the priorities of the federal government? (EQ1)

  • How does it support the current goals of EC?
  • How does it support the current goals of Agriculture and Agri-Food Canada (AAFC)?
  • How does it align with the Next Generation of the Agricultural Policy Framework: Growing Forward?

Design and Delivery

3. Based on your experience with NAESI, how effective is the design of the initiative? (EQ2)

  • Are the themes (air, biodiversity, pesticides, and water) appropriate? Why or why not? What other themes, if any, should NAESI include?
  • How well integrated are the four themes? What other opportunities for integration exist?
  • What are the pros and cons of developing ideal performance standards (IPS) and achievable performance standards (APS)?
  • How do science results support the development of agri-environmental standards?

4. To your knowledge, was NAESI (that is, the initiative as a whole and not the individual standards) implemented as intended? (EQ3)

  • Were the implementation steps well designed and executed on time and within budget?
  • What factors contributed to the successful implementation of the initiative?
  • What factors impeded the successful implementation of the initiative? How were these factors addressed?

5. Now think about the governance structure for NAESI. (EQ4)

  • Are roles and responsibilities of EC and AAFC clearly defined, accepted, and understood?
  • To what extent have EC and AAFC acted upon their responsibilities?
  • To what extent have EC and AAFC shown leadership on NAESI projects?
  • What are the strengths and weaknesses of EC and AAFC’s partnership for this initiative? What opportunities exist to strengthen the partnership?

6. Please assess the effectiveness of processes used to manage: (EQ4)

  • Internal communications within EC and AAFC
  • Communications between EC and AAFC 
  • External communications with provincial governments, producer associations, non-profit organizations, etc.

What improvements should be undertaken, if any?

7. From your point of view, what have been the best practices and key lessons learned relating to the design and delivery of NAESI? (EQ2/3)

  • What have been the strengths and best practices in the design and delivery of the initiative? What has worked well?
  • What have been the challenges or limitations? What has not worked well?
  • What improvements would you suggest to the design and delivery of NAESI?

Success

8. Based on your experience, to what extent have each of the following intended immediate and intermediate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ5)

  • Increased awareness and understanding of the relationships and linkages between agriculture and the environment
  • The incorporation of science results into standards development
  • Development of methods for ideal and achievable performance standards (Prompt: Have an appropriate number of standards been developed? How effective are the standards that have been developed?)
  • The development and distribution of technology transfer and communications support materials (Prompt: What types of information packages have been created? Who have they been distributed to?)
  • Increased awareness and understanding of NAESI results and standards

9. Based on your experience, to what extent have each of the following intended ultimate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ6)

  • Development of ideal and achievable standards that are national, transparent, and coherent with existing roles and responsibilities (regulations) across government and non-government sectors for the following themes:
  1. Air (particulate matter/ammonia)
  2. Biodiversity (habitat conservation)
  3. Pesticides (high risk, commodity) (Prompt: For what percentage of high priority pesticides have standards been developed?)
  4. Water (nutrients, pathogens, sediment, instream flow needs, quantity/conservation)

10. In your view, was an appropriate level of effort devoted to developing standards for each of the theme areas (air, biodiversity, pesticides, and water)? For each theme… (EQ6)

  1. Was an appropriate suite of standards developed? (Prompt: Do the suites comprise the right number and type [ideal performance standards versus achievable performance standards] of standards?) (EQ2)
  2. To what extent does the suite of standards reflect the priorities of the theme?

11. How useful are the set of standards that were developed through NAESI? (EQ7)

  1. How can the standards developed be used? 
  2. To what extent can the standards be implemented? When can they be implemented? What programs are in place to facilitate their implementation? What additional programs are needed?
  3. What synergies across other agri-environmental programs has NAESI facilitated?
  4. To what extent have the results from NAESI been leveraged to be useful for other activities (e.g., other regulatory processes)?

Cost Effectiveness and Alternatives

12. Are you aware of any other government programs or initiatives that have similar objectives to those of NAESI? If yes, please specify. (EQ8)

  1. How is NAESI similar to these programs or initiatives? How is it different?
  2. Are any efforts made to avoid duplication of effort and facilitate the complementarity of NAESI and other government programs or initiatives? If yes, please describe.

13. In your view, has NAESI provided good value for money? (EQ9)

  1. What features of NAESI are cost-effective? What features are cost-ineffective?
  2. How could the cost-effectiveness of NAESI be improved?
  3. In your view, is NAESI the best approach or are there alternative approaches to developing national agri-environmental performance standards that would achieve the same or better results?

14. Do you have any final comments about NAESI?

Thank you for your cooperation.

Overview of the National Agri-Environmental Standards Initiative

The National Agri-Environmental Standards Initiative (NAESI) is a four-year, $25 million program. Its objective is to develop national environmental performance standards for agricultural production. These standards will, either qualitatively or quantitatively, establish the degree of desired environmental quality.

The program’s anticipated immediate and intermediate outcomes include:

  • Policy-makers’ use of science-based standards to inform decision-making
  • Linkage of standards with other initiatives such as the National Agri-Environmental Health Analysis and Reporting Program (NAHARP) and the Watershed Evaluation of Beneficial Management Practices (WEBs) for the purposes of program monitoring and performance measurement.

EC and AAFC are developing two types of standards: Ideal Performance Standards, which specify the desired level of environmental state needed to maintain ecosystem health, and Achievable Performance Standards, which specify the level of environmental quality that can realistically be achieved using currently available and recommended beneficial management practices.

Standards are being developed in four areas: 

  • Water (aquatic ecosystem health): nutrients, sediments, and instream flow needs
  • Biodiversity (aquatic and terrestrial ecosystem health): quality and quantity of habitat to support key components of biodiversity (survival, reproduction) in aquatic and terrestrial ecosystems
  • Pesticides (aquatic and terrestrial ecosystem health): reduce the risk of pesticides
  • Air (human health): farm-level ammonia emissions. 

EVALUATION OF THE NATIONAL AGRI-ENVIRONMENTAL STANDARDS INITIATIVE

INTERVIEW GUIDE
PROVINCIAL GOVERNMENT REPRESENTATIVES

Introduction

The Audit and Evaluation Branch of Environment Canada (EC) is conducting an evaluation of the National Agri-Environmental Standards Initiative (NAESI). A brief overview of the program is appended to this guide. The objective of this evaluation is to assess issues related to the program’s relevance, design and delivery, success, and cost-effectiveness. 

EC has hired PRA Inc., a private research firm, to conduct the evaluation, which comprises a document review and key informant interviews. The questions below serve to guide the interview process. If you are unable to answer a specific question or it does not apply to you, please tell the interviewer and we will skip that question. Please note that your responses will not be linked to you in the evaluation report.

1. Please briefly describe your relationship to and/or involvement with NAESI.

  1. For how long have you had some involvement with the initiative?
  2. With which themes (air, biodiversity, pesticides, and water)/standards have you been most involved? What were your specific roles and responsibilities?
  3. What stakeholders (federal or provincial governments, producer organizations, non-profit organizations, etc.) have you worked with as part of this initiative?

Design and Delivery

2. Based on your experience with NAESI, how effective is the design of the initiative? (EQ2)

  1. Are the themes (air, biodiversity, pesticides, and water) appropriate? Why or why not? What other themes, if any, should NAESI include?
  2. How well integrated are the four themes? What other opportunities for integration exist?
  3. What are the pros and cons of developing ideal performance standards (IPS) and achievable performance standards (APS)?
  4. How do science results support the development of agri-environmental standards? 

3. To your knowledge, was NAESI (that is, the initiative as a whole and not the individual standards) implemented as intended? (EQ3)

  1. Were the implementation steps well designed and executed on time and within budget?
  2. What factors contributed to the successful implementation of the initiative?
  3. What factors impeded the successful implementation of the initiative? How were these factors addressed?

4. From your point of view, what have been the best practices and key lessons learned relating to the design and delivery of NAESI? (EQ2/3)

  1. What have been the strengths and best practices in the design and delivery of the initiative? What has worked well?
  2. What have been the challenges or limitations? What has not worked well?
  3. What improvements would you suggest to the design and delivery of NAESI?

Success

5. Based on your experience, to what extent have each of the following intended immediate and intermediate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ5)

  1. Increased awareness and understanding of the relationships and linkages between agriculture and the environment
  2. The incorporation of science results into standards development
  3. Development of methods for ideal and achievable performance standards (Prompt: Have an appropriate number of standards been developed? How effective are the standards that have been developed?)
  4. The development and distribution of technology transfer and communications support materials (Prompt: What types of information packages have been created? Who have they been distributed to?)
  5. Increased awareness and understanding of NAESI results and standards

6. Based on your experience, to what extent have each of the following intended ultimate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ6)

  1. Development of ideal and achievable standards that are national, transparent, and coherent with existing roles and responsibilities (regulations) across government and non-government sectors for the following themes:
  1. Air (particulate matter/ammonia
  2. Biodiversity (habitat conservation)
  3. Pesticides (high risk, commodity) (Prompt: For what percentage of high priority pesticides have standards been developed?)
  4. Water (nutrients, pathogens, sediment, instream flow needs, quantity/conservation)

7. In your view, was an appropriate level of effort devoted to developing standards for each of the theme areas (air, biodiversity, pesticides, and water)? For each theme… (EQ6)

  1. Was an appropriate suite of standards developed? (Prompt: Do the suites comprise the right number and type [ideal performance standards versus achievable performance standards] of standards?) (EQ2)
  2. To what extent does the suite of standards reflect the priorities of the theme?

8. How useful are the set of standards that were developed through NAESI? (EQ7)

  1. How can the standards developed be used? 
  2. To what extent can the standards be implemented? When can they be implemented? What programs are in place to facilitate their implementation? What additional programs are needed?
  3. What synergies across other agri-environmental programs has NAESI facilitated?
  4. To what extent have the results from NAESI been leveraged to be useful for other activities (e.g., other regulatory processes)?

Cost Effectiveness and Alternatives

9. Are you aware of any other government programs or initiatives that have similar objectives to those of NAESI? If yes, please specify. (EQ8)

  1. How is NAESI similar to these programs or initiatives? How is it different?
  2. Are any efforts made to avoid duplication of effort and facilitate the complementarity of NAESI and other government programs or initiatives? If yes, please describe.

10. In your view, has NAESI provided good value for money? (EQ9)

  1. What features of NAESI are cost-effective? What features are cost-ineffective?
  2. How could the cost-effectiveness of NAESI be improved?
  3. In your view, is NAESI the best approach or are there alternative approaches to developing national agri-environmental performance standards that would achieve the same or better results?

11. Do you have any final comments about NAESI?

Thank you for your cooperation.

Overview of the National Agri-Environmental Standards Initiative

(Repeated from previous guide)

EVALUATION OF THE NATIONAL AGRI-ENVIRONMENTAL STANDARDS INITIATIVE

INTERVIEW GUIDE
PRODUCER ASSOCIATIONS AND NON-PROFIT ORGANIZATIONS

Introduction

The Audit and Evaluation Branch of Environment Canada (EC) is conducting an evaluation of the National Agri-Environmental Standards Initiative (NAESI). A brief overview of the program is appended to this guide. The objective of this evaluation is to assess issues related to the program’s relevance, design and delivery, success, and cost-effectiveness.

EC has hired PRA Inc., a private research firm, to conduct the evaluation, which comprises a document review and key informant interviews. The questions below serve to guide the interview process. If you are unable to answer a specific question or it does not apply to you, please tell the interviewer and we will skip that question. Please note that your responses will not be linked to you in the evaluation report.

1. Please briefly describe your relationship to and/or involvement with NAESI.

  1. For how long have you had some involvement with the initiative?
  2. With which themes (air, biodiversity, pesticides, and water)/standards have you been most involved? What were your specific roles and responsibilities?
  3. What stakeholders (federal or provincial governments, producer organizations, non-profit organizations, etc.) have you worked with as part of this initiative?

Design and Delivery

2. To your knowledge, was NAESI (that is, the initiative as a whole and not the individual standards), implemented as intended? (EQ3)

  1. Were the implementation steps well designed and executed on time and within budget?
  2. What factors contributed to the successful implementation of the initiative?
  3. What factors impeded the successful implementation of the initiative? How were these factors addressed?

3. From your point of view, what have been the best practices and key lessons learned relating to the design and delivery of NAESI? (EQ2/3)

  1. What have been the strengths and best practices in the design and delivery of the initiative? What has worked well?
  2. What have been the challenges or limitations? What has not worked well?
  3. What improvements would you suggest to the design and delivery of NAESI

Success

4. Based on your experience, to what extent have each of the following intended immediate and intermediate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ5)

  1. Increased awareness and understanding of the relationships and linkages between agriculture and the environment
  2. The incorporation of science results into standards development
  3. Development of methods for ideal and achievable performance standards (Prompt: Have an appropriate number of standards been developed? How effective are the standards that have been developed?)
  4. The development and distribution of technology transfer and communications support materials (Prompt: What types of information packages have been created? Who have they been distributed to?)
  5. Increased awareness and understanding of NAESI results and standards

5. Based on your experience, to what extent have each of the following intended ultimate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ6)

  1. Development of ideal and achievable standards that are national, transparent, and coherent with existing roles and responsibilities (regulations) across government and non-government sectors for the following themes:
    1. Air (particulate matter/ammonia)
    2. Biodiversity (habitat conservation)
    3. Pesticides (high risk, commodity) (Prompt: For what percentage of high priority pesticides have standards been developed?)
    4. Water (nutrients, pathogens, sediment, instream flow needs, quantity/conservation)

6. In your view, was an appropriate level of effort devoted to developing standards for each of the theme areas (air, biodiversity, pesticides, and water)? For each theme… (EQ6)

  1. Was an appropriate suite of standards developed? (Prompt: Do the suites comprise the right number and type [ideal performance standards versus achievable performance standards] of standards?) (EQ2)
  2. To what extent does the suite of standards reflect the priorities of the theme?

7. How useful are the set of standards that were developed through NAESI? (EQ7)

  1. How can the standards developed be used? 
  2. To what extent can the standards be implemented? When can they be implemented? What programs are in place to facilitate their implementation? What additional programs are needed?
  3. What synergies across other agri-environmental programs has NAESI facilitated?
  4. To what extent have the results from NAESI been leveraged to be useful for other activities (e.g., other regulatory processes)?

Cost Effectiveness and Alternatives

8. Are you aware of any other government programs or initiatives that have similar objectives to those of NAESI? If yes, please specify. (EQ8)

  1. How is NAESI similar to these programs or initiatives? How is it different?
  2. Are any efforts made to avoid duplication of effort and facilitate the complementarity of NAESI and other government programs or initiatives? If yes, please describe. 

9. In your view, has NAESI provided good value for money? (EQ9)

  1. What features of NAESI are cost-effective? What features are cost-ineffective?
  2. How could the cost-effectiveness of NAESI be improved?
  3. In your view, is NAESI the best approach or are there alternative approaches to developing national agri-environmental performance standards that would achieve the same or better results?

10. Do you have any final comments about NAESI?

Thank you for your cooperation.

Overview of the National Agri-Environmental Standards Initiative

(Repeated from previous guide)

Top of Page

Appendix 3

References

Agriculture and Agri-food Canada..2003. Federal-Provincial-Territorial Framework Agreement on Agricultural and Agri-Food Policy for the Twenty-First Century. Available from http://www4.agr.gc.ca/AAFC-AAC/
display-afficher.do?id=1201110596840&lang=eng#s1
(accessed March 2, 2009).

Agriculture and Agri-Food Canada. n.d. 2007–2008 Report on Plans and Priorities. Available from http://www.TBS-sct.gc.ca/RPP/0708/
AAFC-aac/AAFC-aac-eng.pdf (PDF)
(accessed March 11, 2009).

Agriculture and Agri-Food Canada. 2007. Growing Forward: A Federal-Provincial-Territorial Framework Agreement on Agriculture, Agri-Food and Agri-Based Products Policy. Available from http://www4.agr.gc.ca/AAFC-AAC/
display-afficher.do?id=1224167497452&lang=eng
(accessed March 11, 2009).

Belanger, D. 2008. February 5–6, 2008 Technical Meeting Report. National Agri-Environmental Standards Initiative Technical Series Report No. 4-62. 230 p.

Bowerman, M., L. Maclean, S. Villeneuve, and E. S. Roberts. 2009. National Agri-Environmental Standards Initiative Overarching Report. National Agri-Environmental Standards Initiative Synthesis Report No. 1. Environment Canada. Gatineau, Quebec. 239 p.

Canada. 2003. Memorandum of Understanding: Agricultural Policy Framework Initiatives Development (cited in Bowerman et al., 2009). December 2003.

Canada. 2007a Speech from the Throne. Available from: http://www.ctv.ca/servlet/ArticleNews/
story/CTVNews/20071016/thronespeech_SIDEBARS_071016/20071016/
(accessed March 3, 2009). 

Canada. 2007b. National Agri-Environmental Standards Initiative Brochure. Environment Canada and Agriculture and Agri-Food Canada.

Canada. 2007c. National Agri-Environmental Standards Initiative, Newsletter No. 2. Environment Canada and Agriculture and Agri-Food Canada. July 2007.

Carona Designs Inc. 2009. National Agri-Environmental Standards Initiative (NAESI) Final Stakeholder Workshop. January 7–8, 2009. National Agri-Environmental Standards Initiative. Environment Canada. 134 p.

Ellis, D. 2006. National Agri-Environmental Standards Initiative (NAESI) Stakeholder Consultation Workshop--Proceedings. National Agri-Environmental Standards Initiative Technical Series Report No. 2-71. 81 p.

Environment Canada. 2005. Results-based Management and Accountability Framework (RMAF) and Risk Based Accountability Framework (RBAF) for the National Agri-Environmental Standards Initiative (NAESI). June 1, 2005.

Environment Canada. 2007. 2007-2008 Report on Plans and Priorities. Available from http://www.tbs-sct.gc.ca/rpp/0708/
ec-ec/ec-ec-eng.pdf (PDF)
(accessed March 3, 2009)

Environment Canada. 2008. Evaluation Plan for the National Agri-Environmental Standards Initiative (NAESI).

Lefebvre, A. 2006. National Agri-Environmental Standards Initiative (NAESI). Presentation at Environment Chapter Leads Meeting. Ottawa, Ontario. February 8–9, 2006.

Roberts, E. 2006. The Program. Presentation at NAESI Stakeholder Consultation Workshop Toronto, Ontario. March 21, 2006

Roberts, E. 2009. Overview of NAESI. Presentation at final NAESI Stakeholder Workshop Winnipeg, Manitoba. January 7, 2009.

Top of Page

Appendix 4

NAESI Communications Activities

The initiative’s Communications Strategy identified the responsibilities of Environment Canada (EC) and Agriculture and Agri-Food Canada (AAFC) for communicating the purpose, methodologies, results and uses of the standards, and was designed to guide managers, the Secretariat, regional coordinators, and theme and project leads. 

Roles and Responsibilities Related to Communications

Stakeholder roles and responsibilities pertaining to communications activities were specified in the NAESI Communications Strategy, which was an appendix to the Overarching Report, and are outlined in Table 4-1. 

Table 4-1. Roles and responsibilities relating to National Agri-Environmental Standards Initiative NAESI communications activities
ParticipantRoles and responsibilities
Secretariat
  • Coordinate implementation of communication strategy
  • Liaise with AAFC
  • Ensure linkages between NAESI standards and other Agricultural Policy Framework (APF) environment programs
Regional coordinators
  • Facilitate outreach and consultation activities during standards development
Theme leads
  • Communicate scientific findings at meetings and conferences
  • Strengthen linkages with other APF programs
  • Represent EC on other projects and committees
Researchers
  • Communicate findings at meetings and scientific conferences

AAFC liaison 

(Agri-Environmental Policy Bureau)

  • Liaison between NAESI and AAFC
  • Share information about NAESI within the department
  • Review communications pieces
  • Assist in coordinating communication strategy
  • Ensure NAESI website is up-to-date

 

Source: Bowerman et al. 2009, pp. 187–195

NAESI Communications Products

The Communications Strategy outlined current and ongoing federal and external communications activities, as well as tools used to conduct activities, which included stakeholder consultations, the NAESI brochure and website, the annual NAESI Technical Series, and the newsletter. The Strategy described each tool, outlined its objectives and identified its target audience, noting that communications tools originated from the Secretariat and that communications products intended for the public were developed with and approved by the AAFC liaison.

Through the four years, two stakeholder consultation workshops were held. The first, held in Toronto in March 2006, focused on increasing the understanding about NAESI, identifying the uses of the standards among participants, disseminating information on programs into which the standards could link, and developing recommendations to address participant concerns related to communication and standards use. The NAESI brochure and newsletter were created in response to stakeholder feedback from this first workshop. 

The second stakeholder workshop, held in Winnipeg in January 2009, provided an opportunity for EC and AAFC representatives to present the draft standards, discuss limitations and gaps in the science, and identify opportunities to use the standards to inform decision-making in agriculture (Carona Designs Ltd. 2009).

The NAESI brochure (Canada 2007b), introduced in April 2007, provided an overview of NAESI for stakeholders and the public, including partners. It outlined the objectives of NAESI, potential uses of standards, as well as links to other APF programs. The newsletter, introduced in January 2007, provided regular updates to stakeholders regarding the findings and presented examples of how standards could be employed. In total, four issues of the newsletter were published between January 2007 and October 2008. The NAESI Technical Series, intended for the informed public and scientific community, produced 247 reports released in four annual issues between 2005 and 2008. The reports detailed the program’s technical findings, which formed the foundation of the standards. 

The Communications Strategy then outlined the dissemination of NAESI results in particular settings. The Secretariat reviewed the messaging and policy implications in the materials and incorporated any comments from AAFC. In accordance with the Strategy, EC summarized work done in each thematic area in a series of synthesis reports, including discussions of key findings, methodology, sampling locations and draft standards.

The Strategy outlined the peer review process, which was completed by March 2008. This process involved the review of each NAESI technical synthesis report, different from a technical report, by a minimum of three independent researchers, with the process overseen by an independent evaluator. The submission of the NAESI technical synthesis to the peer review process ensured the scientific validity of the standards.  

The value of the expertise that resided within NAESI’s stakeholder groups was recognized by EC and AAFC, and a stakeholder review was carried out in conjunction with the 2009 stakeholder workshop. This review provided involved stakeholder organizations with an opportunity to ask questions and provide technical advice and recommendations to report authors.

Top of Page

Appendix 5

NAESI Timeline

Timeline for National Agri-Environmental Standards Initiative NAESI events and deliverables for standards
DateEvents and deliverables
NAESI Newsletter No. 1
(December 2006)
March 21–22, 2006: Environment Canada (EC) and Agriculture and Agri-Food Canada (AAFC) held a workshop to introduce NAESI to stakeholders.
Air:Nine projects underway to improve understanding of ammonia released from agricultural sources and its relation to particulate matter. Team reviewing other national and international approaches to determine whether NAESI standard can be developed.
Biodiversity: Nine projects underway to develop three-tiered terrestrial and riparian habitat quantity and quality standards.
Pesticides: Nine projects underway to quantify the risks of agricultural pesticides to ecosystem and human health. Ideal performance standards (IPS) being developed for 10 additional pesticides. Also developing a spray advisory meteorological standard to assist farmers with pesticide application.
NAESI Newsletter No. 2
(July 2007)
Water: Twenty-eight projects underway to develop standards that will protect streams and coastal waters from the harmful effects of excessive nutrients, sediments, microbial pathogens and unsustainable water use. Water quality standards being developed indicating low numbers of waterborne pathogens for various farming areas.
December 2006: NAESI annual technical workshop. Team leads met to present and provide updates on their projects, and to plan for the final two years of NAESI.
Established two-phase peer-review process to be conducted by an independent evaluator. Will include an academic technical review planned to start September 2007 and a formal partner and stakeholder technical review planned for March 2008.
NAESI brochure published to provide non-technical, descriptive program material to the general public.
Air: The air theme will deliver a suite of regionally specific ammonia standards. By the summer, collection and analysis of field data will be finalized, national ammonia emissions inventory will be completed, and the regional air quality modelling will be carried out. A technical document summarizing the agricultural odour work carried out during NAESI will also be prepared. 
Biodiversity: Finalizing tier 1 generalized habitat standards for seven ecozones. Tier 2 standards for four pilot ecoregions nearing completion. Tier 3 standards being tested in four pilot areas. A land and water integration decision-support tool being developed. 
Pesticides: Completing risk-based rankings for pesticides relative to the development of IPS and achievable performance standards (APS). Pilot spray advisory being finalized for Southern Ontario. Proposed project to compare National Agri-Environmental Health Analysis and Reporting Program soil landscape risk determinations to the NAESI watershed approach.
NAESI Newsletter No. 3
(December 2007)
Water:Complete the acquisition of field and existing data for setting water quality and quantity standards. Focus on validating recommended IPS at national and regional levels. Work continues on integrated modelling framework to estimate water balance indicators. The 2007 field season will be used to determine the best tools and techniques for environmental monitoring and performance measurement.
On track to deliver a suite of science-based, peer-reviewed agri-environmental performance standards by April 2008. Total of 85 standards to be delivered to AAFC consisting of IPS and APS.
Air:Recent efforts focused on modelling scenarios to quantify expected air quality improvements that could be achieved through ammonia reductions from agricultural sources. Air theme to produce one synthesis report on regionally specific ammonia standards.
Water: Activities have been spread out over five sub-themes (sediments, nutrients, instream flow needs, pathogens and water availability). Water theme produced six synthesis reports covering 14 standards.
Biodiversity: NAESI biodiversity standards will answer the question “How much habitat is enough?” on several different scales. Biodiversity theme to produce two synthesis reports covering 15 tier 1 standards that will be applied to and refined for each pilot area (tier 2 and 3).
NAESI Newsletter No. 4
(October 2008)
Pesticides: Activity to develop aquatic and terrestrial standards for individual agricultural pesticides, pesticide mixtures and one commodity. Pesticide theme to produce six synthesis reports covering 51 standards.
February 2008:EC hosted AAFC at a workshop in Calgary. After both departments reviewed the synthesis reports, the consensus was that NAESI had fostered successful interdepartmental collaboration that resulted in a number of tangible products and that future collaboration should be pursued.
March 31, 2008: Delivered results of work towards the development of agri-environmental standards to AAFC.
NAESI standards fall under one of four classifications: IPS, APS, guidance, provisional.
NAESI results presented in a series of 15 peer-reviewed synthesis reports that describe the approach used to develop standards, final results and key scientific findings.
Workshop planned for early 2009 to discuss potential application of the NAESI results.
NAESI Workshop 
(January 2009)
Final NAESI Stakeholder Workshop Proceedings are available (Carona Designs Inc. 2009)
NAESI synthesis reports
(April 2009)
Final NAESI products (set of 16 synthesis reports) were released.

Top of Page

Appendix 6

NAESI Standards by Thematic Area

Water Standards

As shown in Table 6-1, the 16 standards for the water theme were developed across five topics, including nutrients (toxicity and eutrophication), sediments, pathogens, water availability and instream flow needs. Three of the standards were classified as provisional standards, including the numerical or descriptive benchmark for pathogens, and the methodological and guidance-based standards for water availability and instream flow needs.

Table 6-1. Standards in the water theme
Sub-themeAchievable Performance StandardsIdeal Performance StandardsTotal
Numerical or descriptive benchmarkMethodological or guidance
Nutrients    
– toxicity44
– eutrophication235
Sediments134
Pathogens(provisional)11
Water availability(provisional) 11
Instream flow needs(provisional) 11
Total3(1 provisional) 11 (2 provisional) 2 16
Source : Bowerman et al. 2009, sec. 3.0; Roberts 2009

 

Biodiversity Standards

As shown in Table 6-2, a total of 17 standards were developed for the biodiversity theme, across six topics: natural areas, riparian ecosystems, forest ecosystems, grasslands ecosystems, wetlands ecosystems and anthropogenic areas. All of the standards were classified as numerical or descriptive benchmarks.

Table 6-2: Standards in the biodiversity theme
Sub-themeAchievable Performance StandardsIdeal Performance StandardsTotal
Numerical or descriptive benchmarkMethodological or guidance
Natural areas 4 4
Riparian ecosystems33
Forest ecosystems22
Grasslands ecosystems44
Wetlands ecosystems33
Anthropogenic areas11
Total1717
Source: Bowerman et al. 2009, sec. 3.0; Roberts 2009

 

Pesticide Standards

As shown in Table 6-3, 64 standards were developed for the pesticide theme, across six topics: IPS, APS, risk-based standards, mixtures standards, commodity-based standards and meteorological standards.  

Table 6-3. Standards in the pesticide theme
Sub-themeAchievable Performance StandardsIdeal Performance StandardsTotal
Numerical or descriptive benchmarkMethodological or guidance
IPS4141
APS1212
Risk-based standards88
Mixtures standards11
Commodity-based standards11
Meteorological standards11
Total12411164
Source: Bowerman et al. 2009, sec. 3.0; Roberts 2009

 

Air Standards

As shown in Table 6-4, one standard was developed for the air theme, associated with ammonia as it relates to the formation of airborne particulate matter. 

Table 6-4. Standards in the air theme
Sub-themeAchievable Performance StandardsIdeal Performance StandardsTotal
Numerical or descriptive benchmark Methodological or guidance
Ammonia (particulate matter)1 1
Total11
Source: Bowerman et al. 2009, sec. 4.0; Roberts 2009

Top of Page

Appendix 7

NAESI Outputs

NAESI Outputs for standards by theme and sub-theme
Theme and sub-themeOutputs
Water
Nutrients- eutrophicationAnalyses of nitrogen and phosphorus (N and P) concentrations in samples from more than 200 water-quality stations across Canada.
 Detailed ecological studies of approximately 70 stream sites and 200 medium and large rivers.
 Recommendations for ideal performance standards (IPS) for N and P levels for streams and agricultural watersheds.
 New approach presented for assessing physical susceptibility to nutrient enrichment for coastal waters.
Nutrients- toxicityDerived short- and long-term nitrate IPS values for marine and freshwater environments.
Sediments Report on physical and biological IPS values for total suspended sediments, turbidity and deposited-sediment thresholds predicted to be protective of environmental quality.
 Established monitoring programs in six provinces to expand the spatial and temporal range for standards development and to collect the necessary biological data to field test the IPS values.
 Study aimed at deposit-sediment standard development; first comprehensive effort at performance-standard development for agricultural streams in Canada.
 Non-point source water quality modelling was used to develop achievable performance standards (APS) for total suspended sediments in streams of two agricultural watersheds.
Pathogens Samples analyzed from four agricultural watersheds to develop IPS for waterborne pathogens.
 Four watersheds (27 sites) analyzed for five pathogens (campylobacter, cryptosporidium, giardia, salmonella and E. coli) and other water quality indicators.
 Evaluated existing Canadian water quality standards to determine whether they would be a suitable NAESI standard.
 Developed provisional NAESI IPS for agricultural sites on small streams and on large streams or rivers.
Instream flow needs Examined the geographical variability of air temperature, precipitation and flow regimes across the agricultural region of Canada.
 Development of the Canadian Ecological Flow Index, which permits watershed assessment of impairment due to loss of flow habitat.
Water availability Development of water availability indicators using a coupled hydrometeorological modelling system, together with a data-assimilation system, has been useful for describing the water cycle at the watershed scale.
Biodiversity
Tier 1 generalized habitat-based standards Produced NAESI tier-1 agri-environmental standards for natural areas, grasslands, forests, riparian areas, wetlands, anthropogenic areas and species at risk.
Tier 2 landscape-specific habitat-based standards Report outlines a process for developing habitat-based standards under the NAESI biodiversity theme.
 Case study used to demonstrate the application of the methodology and to measure the impact of habitat quantity on water quality standards.
Pesticides
Ideal performance standardsSynthesis of the development of IPS for priority pesticides currently in use in Canada (20 freshwater, 1 sediment).
IPS provides guidance for acute and chronic pesticide exposure when possible.
Achievable performance standardsDevelopment of APS for eight pesticides under NAESI.
Inventory of existing hydrological models of pesticide transport at the watershed level and the selection of three relevant to the needs of the study.
Models applied to Beaurivage River watershed to carry out an initial classification of the pesticides based on APS.
Case study results applied to five additional watersheds chosen by Environment Canada.
Risk-based standardsAssembled empirical database of terrestrial and aquatic field studies to generalize from lab-based toxicity indices to actual environmental sectors potentially affected by pesticide use.
Report presents the believed best measurement instruments to gauge the relative environmental impact of various pesticide treatments.
Helps identify pesticide products in need of replacement or mitigation.
Commodity-based standards Development of commodity-based ideal performance standards (CB-IPS) that assess impacts to aquatic environments from exposure to mixtures of chemicals related to a specific commodity.
CB-IPS approach applied to two watersheds and to historical data from the top five potato-producing provinces.
CB-IPS is a practical tool to identify particular watersheds that may be at risk from commodity pesticide mixtures, but only applies when that commodity is the dominant crop grown in that watershed.
Mixtures standardsSynthesis of efforts to develop NAESI IPS for pesticide mixtures.
Protocol can be used to predict expected joint toxicity effects of mixtures, with some assumptions.
Can be used to identify watersheds potentially at risk from pesticide mixtures, and to help guide agricultural stewardship and water quality monitoring programs.
Meteorological standardsDeveloped a spray advisory to help farmers determine when meteorological conditions are appropriate for spraying pesticides.
Users in an agricultural region of Ontario were surveyed regarding the usefulness and application of the advisory; 91% found it useful in making spraying decisions.
Air
Ammonia NAESI research significantly advanced understanding of agricultural NH3 emissions.
Revised 2002 Canadian national agricultural ammonia emissions inventory to incorporate geographic and temporal variability of emissions.
Sensitivity of particulate matter concentrations to theoretical reductions in NH3 emissions assessed for six agricultural regions across Canada.
Regionally specific NH3 standards developed for five agriculturally intensive areas in Canada.

Top of Page

Appendix 8

Summary of Findings

Evaluation question (EQ)OutcomeAchievedSome progressLittle progress to date
QE1   
QE2  X 
QE3  X 
QE4   
QE5 Immediate and intermediate outcome a~√  
 Immediate and intermediate outcome b  
 intermediate outcome c  
QE6Ultimate outcome  
QE7   X*
QE8   
QE9   

 

Notes

~√ Although there is compelling subjective evidence that the initiative has done well with respect to this evaluation question, a complete assessment cannot be done due to a lack of performance data.

X* Although early progress has been made in identifying potential uses for some of the NAESI standards, it is not clear whether and how many of the standards developed will be used or implemented.

 

1 - In actual fact, funds were not provided to Enivronment Canada until April 1, 2004. Work commenced on that date as well. See “2.3 Financial Resources,” below, for more information. Back to top

2 - In actual fact, funds were not provided to Environment Canada until April 1, 2004. Work commenced on that date as well. See “2.3 Financial Resources,” below, for additional details. Back to top

3 - This section draws from Environment Canada (2005), Roberts (2006, 2009) and Bowerman et al. (2009). Back to top

4 - Under this framework, Environment Canada is responsible for the performance standards, while Agriculture and Agri-Food Canada is responsible for the process standards. Back to top

5 - While the definition of ideal performance standards is unchanged from the accountability framework in the Overarching Report, the definition for acheiveable performance standards appears to have been slightly revised to read, “environmental conditions that can realistically be achieved using a variety of land management scenarios including currently available and recommended [beneficial management practices]” (Bowerman et al. 2009, sec. 1.6). Back to top

Date modified: