Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2008-2009 > Evaluation of the National Agri-Environmental Standards Initiative

Evaluation of the National Agri-Environmental Standards Initiative

Final Report

July 20, 2009

( Word Version , 1.60 Mb )

Report Clearance Steps

Acronyms Used in the Report

Acknowledgements

The Evaluation Project Team, led by Michael Callahan under the direction of Shelley Borys, would like to thank those individuals who contributed to this project, particularly all the interviewees, who provided crucial insights and comments. We would also like to thank the members of the Joint Evaluation Committee for their guidance and support: Elizabeth Roberts, from Environment Canada, and Rohit Tuteja and Robin MacKay, from Agriculture and Agri-Food Canada.

The evaluation was conducted by PRA Inc., which prepared this report along with the Evaluation Division, Audit and Evaluation Branch, Environment Canada.

Contents

Executive Summary

1.0 Introduction

1.1 Definition of Standards

1.2 Outline of the Report

2.0 Overview of the Initiative

2.1 Background

2.2 Objectives and Expected Results

2.3 Financial Resources

2.4 Initiative Reach – Stakeholders and Partner

2.5 Governance Structure

2.6 Roles and Responsibilities

2.7 Performance Reporting

2.8 Initiative Outputs and Outcomes

3.0 Evaluation Design

3.1 Purpose and Scope of Evaluation

3.2 Evaluation Approach and Methodology

3.3 Limitations of the Evaluation

3.4 Evaluation Ratings

4.0 Findings

4.1 Relevance

4.2 Design and Delivery

4.3 Success

4.4 Cost-Effectiveness

5.0 Conclusions

5.1 Main Findings

5.2 Key Lessons Learned

Appendix 1 Evaluation Issues and Questions

Appendix 2 Interview Guides

Appendix 3 References

Appendix 4 NAESI Communications Activities

Appendix 5 NAESI Timeline

Appendix 6 NAESI Standards by Thematic Area

Appendix 7 NAESI Outputs

Appendix 8 Summary of Findings

Executive Summary

Introduction

Science plays two key roles in mitigating the external effects of farming. First, science identifies the limits of soil, water and air to absorb farming activity. Second, science provides the theoretical framework for designing and evaluating mitigating strategies, commonly termed beneficial management practices.

Led by Environment Canada, in partnership with Agriculture and Agri-Food Canada, the National Agri-Environmental Standards Initiative (NAESI) received $25 million through a Memorandum of Understanding between Agriculture and Agri-Food Canada and Treasury Board of Canada Secretariat for five years (2003–2004 to 2007–2008) to “develop national environmental performance standards for agricultural production” (Lefebvre 2006).¹ Focusing on four thematic areas—water, biodiversity, pesticides and air—these standards are intended to support, qualitatively and quantitatively, the development of non-regulatory agri-environmental actions under the Agricultural Policy Framework and other initiatives by government and environmental non-governmental organizations.

Environment Canada held a final stakeholder meeting in January 2009 and will deliver these performance standards to Agriculture and Agri-Food Canada in 2009 for implementation. The NAESI Memorandum of Understanding required that an evaluation of the initiative be done.

The Evaluation Division, Audit and Evaluation Branch, of Environment Canada oversaw the evaluation. The Department commissioned PRA Inc., an independent research firm, to carry out the necessary data collection and analysis.

Evaluation Issues

The evaluation examined the following issues:

• Relevance: Was the initiative consistent with departmental and government-wide priorities?

• Design and delivery: Was the initiative designed and delivered in the best possible way?

• Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?

• Cost-effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?

Methodology

Since NAESI has ended and is not being considered for renewal, the evaluation was limited in scale, relying on two qualitative lines of evidence—interviews and document review. 

• The study team interviewed 18 representatives from Environment Canada, Agriculture and Agri-Food Canada, other federal government departments, provincial governments and other stakeholders (e.g. environmental non-governmental organizations and industry). 
• The study team reviewed relevant documents and literature to compile a profile of the initiative and to answer specific evaluation questions. Environment Canada provided most of the documents for review.

Overview of Findings and Conclusions

NAESI aligned with Environment Canada’s strategic outcomes and program areas. It was also part of the Environment Pillar of Agriculture and Agri-Food Canada’s Agricultural Policy Framework and is logically consistent with Growing Forward, the successor to the to this framework. Interviewees said that the four thematic areas selected to categorize the standards were appropriate for agri-environmental programming.

The partnership between Environment Canada and Agriculture and Agri-Food Canada was seen as reflecting good design practice, as was the practice of involving stakeholders such as industry groups. All interviewees supported the governance structure and the appointment of a single designated program manager, seeing these as effective delivery processes.  

NAESI increased scientific understanding of the relationship between agriculture and the environment, and developed 98 standards. While some variation exists in the number and completeness of the standards developed across the four thematic areas, and work remains to be completed in some areas, interviewees said that a significant body of research has been synthesized to support agri-environmental policy and programming.

The administrative costs of NAESI were low, and the initiative did not duplicate existing programming in Canada. The initiative linked to important federal agri-environmental initiatives, and most interviewees said that the initiative generated state-of-the art scientific information; however, it is difficult to judge the ultimate value of NAESI, since it is unclear whether and how the standards developed will be implemented.

The realization of the long-term strategic outcomes of NAESI (e.g. improved stewardship by agricultural producers of land, water, air and biodiversity) will require the development of an explicit implementation and adoption strategy to integrate these standards into emerging agri-environmental programming and policy, especially within Growing Forward.

Findings by Evaluation Issue

Lessons Learned

Since NAESI has ended, the focus of this evaluation was on lessons learned as opposed to recommendations for improvement. The following lessons learned from the experience with NAESI may serve to inform the design of future similar initiatives.

1. Clearly stating the intent and objectives of the initiative at its outset, especially with respect to implementation, increases partner and stakeholder support and buy-in. This also involves ensuring all stakeholders understand the language used in memoranda of understanding and communication materials. Had this happened for this initiative, work on the incomplete standards may have increased and stakeholder implementation of NAESI’s outputs may have been encouraged.
2. A single program manager and point of contact proved instrumental to this horizontal initiative. The manager facilitated interdepartmental communication and coordinated diverse technical initiatives.
3. Maintaining strong relationships and communication with partners is essential not only to the creation of an initiative’s intended outputs but also for the realization of its intended outcomes. For an initiative such as NAESI, the standards developed must be integrated into agri-environmental programming that supports on-farm adoption of beneficial management practices. Agriculture and Agri-Food Canada has primary responsibility for ensuring that NAESI outputs become integrated into the agri-environmental programming under Growing Forward. 
4. The primary objective of NAESI was to develop agri-environmental standards. While NAESI developed national standards, synchronization with existing or planned provincial standards or regulations remains unfinished. Consultation with provincial stakeholders at an earlier stage of the initiative might have enabled NAESI to become integrated with provincial processes. This might also have helped stimulate additional support for the initiative. 
5. Defining an implementation strategy for the outputs is essential for their integration into agri-environmental programming. Although NAESI added to the scientific knowledge base and established standards, it lacked an implementation strategy; therefore, it cannot be said that it has progressed toward its strategic outcomes. In the final analysis, realization of the strategic outcomes will require an implementation plan to promote the use and application of the standards through the agri-environmental programming under Growing Forward.

Top of Page

1.0 Introduction

Science plays two key roles in mitigating the effects of farming. First, it identifies the limits of soil, water and air to absorb farming activity. Second, it provides the theoretical framework for designing and evaluating mitigating strategies, commonly termed beneficial management practices.

Agriculture and Agri-Food Canada’s Agricultural Policy Framework supported concrete improvement in the quality of the environment through the coordinated adoption of beneficial management practices on farms. This framework aimed to increase and improve the use of environmental planning tools and management systems by Canadian farmers.

Approved in February 2003, the Agricultural Policy Framework set out the need to develop a suite of non-regulatory standards for the environmental quality required of agriculture and to validate beneficial management practices. Non-regulatory standards promote environmental practices that reduce agricultural risks, thus benefiting the health and supply of water, the health of soils, the health of air, and the compatibility of biodiversity and agriculture. Non-regulatory standards also provide a benchmark for reporting to Canadians on the results of improved agricultural management practices. 

The National Agri-Environmental Standards Initiative (NAESI), led by Environment Canada in partnership with Agriculture and Agri-Food Canada, supported the role of science in mitigating the external effects of agriculture. NAESI had the following goals:

• to establish non-regulatory national environmental performance standards (with regional application) that define common Environment Canada and Agriculture and Agri-Food Canada goals for the environment;
• to evaluate standards attainable by use of existing environmentally beneficial agricultural production and management practices; and
• to increase understanding of relationships between agriculture and the environment (Agriculture and Agri-Food Canada, n.d.). 

This initiative received $25 million through a Memorandum of Understanding between Agriculture and Agri-Food Canada and Treasury Board of Canada Secretariat for five years (2003–2004 to 2007–2008).² Over the five years, NAESI used this funding to develop environmental performance standards to guide beneficial management practices to be implemented at the farm level. Utilizing funds carried over from 2007–2008, Environment Canada held a final stakeholder meeting in January 2009 to review the performance standards, which it will deliver to Agriculture and Agri-Food Canada in 2009 for implementation. The Memorandum of Understanding required an evaluation of NAESI.

This evaluation assessed the relevance of NAESI and whether and to what extent it achieved its intended outcomes, as well as whether it was designed and delivered in the most appropriate and cost-effective way. Lessons learned from the initiative could serve to inform the design of future similar initiatives. 

A committee of program and evaluation representatives from Environment Canada and Agriculture and Agri-Food Canada guided the planning phase of the evaluation, including developing the interview questions and a list of interviewees. Environment Canada’s Evaluation Division, Audit and Evaluation Branch, oversaw the evaluation and commissioned PRA Inc., an independent research firm, to conduct the required interviews and analysis.

Top of Page

1.1 Definition of Standards

NAESI developed agri-environmental standards in four areas:

• water (aquatic ecosystem health): nutrients, pathogens and instream needs, as well as sediments and water availability;
• biodiversity (aquatic and terrestrial ecosystem health): habitat conservation, including of wetlands, riparian areas and connective corridors;
• pesticides (aquatic and terrestrial ecosystem health): 20 priority pesticides; and 
• air (human health): ammonia emissions. 

Within these four areas, there are two types of standards: ideal performance standards, which “specify the desired level of environmental state needed to maintain ecosystem health,” and achievable performance standards, which “specify the level of environmental quality that can realistically be achieved using currently available and recommended beneficial management practices” (Roberts 2009). Specifically, these standards comprise the following:  

• non-regulatory instruments;
• quantitative or qualitative measures (descriptive benchmarks) of desired environmental qualities that are scientifically defensible and focused on key agri-environmental issues;
• useful tools to inform landowners, decision makers and policy-makers when setting targets for desired degrees of environmental performance for air and water quality, biodiversity and pesticide use in agricultural areas; and
• definitions of maximum concentrations of harmful substances, specified condition of the environment or habitat standards (biodiversity) (Canada 2007b).

Top of Page

1.2 Outline of the Report

This document presents the findings and conclusions of the evaluation. Section 2 provides background information on NAESI. Section 3 describes the purpose of this evaluation and the methodology used. Section 4 presents the evaluation’s findings. Finally, Section 5 presents the conclusions and lessons learned based on the evaluation findings.

Top of Page

2.0 Overview of the Initiative

The purpose of NAESI, which fit under the Environment Pillar of the Agricultural Policy Framework, was to develop “national non-regulatory, science-based agri-environmental performance standards […] to guide agri-environmental decision-making.”³ Environment Canada and Agriculture and Agri-Food Canada signed a Memorandum of Understanding in December 2003 to implement NAESI. Work commenced April 1, 2004, with the program scheduled to end March 31, 2008. (In actual fact, work continued past that date to ensure all products were completed.)

NAESI had two activity areas (see “2.8 Initiative Output and Outcomes,” below). 

• Activity area 1 assessed “risk, priorities, and partnerships for performance standards” for the thematic areas.  
• Activity area 2 developed national standards intended to reduce the impact of the agricultural sector on the environment. 

Work on NAESI was undertaken in stages: Stage 1 (2004–2005) was a scoping exercise to determine those aspects of the environment most affected by agriculture. This first stage supported priority setting, which preceded the development, demonstration and testing of standards in Stage 2 (2005–2008). Stage 3, which took place during the last year of the initiative (2007–2008), involved the conclusion of standards testing and the delivery of results to Agriculture and Agri-Food Canada (Roberts 2006; Bowerman et al. 2009, sec. 1.4). The tangible outputs of the initiative are standards compiled in a series of synthesis reports.

NAESI used a peer review process to review the standards as expressed in the synthesis reports. The peer review process included an academic review overseen by an independent evaluator, followed by a technical review by experts from partner and stakeholder organizations. The process assessed the methodologies used to develop the standards and the standards themselves. In effect, this review process assessed “the science behind standard development rather than issues of policy and implementation” (Bowerman et al. 2009, sec. 2.4). The peer review served as a check on the scientific validity and quality of the outputs.

Top of Page

2.1 Background

NAESI was one of several agricultural programs outlined in the Agricultural Policy Framework, a five-year (2003–2008), $5.2-billion initiative of the Government of Canada and the provincial governments. (The Framework was replaced by Growing Forward; see “4.1 Relevance,” below.) The Framework consisted of five integrated pillars for programming and policy development: Business Risk Management, Food Safety and Quality, Science and Innovation, Environment, and Renewal. NAESI fell under the Environment Pillar (Agriculture and Agri-Food Canada 2003). Policy and programs under the Environment Pillar were allotted $526 million to improve agricultural producers’ stewardship of the soil, water and air, and enhance biodiversity. To this end, the Government of Canada and the provinces agreed to conduct research and development efforts intended to do the following: 

• increase understanding of relationships between agriculture and the environment;
• develop and evaluate environmentally beneficial agricultural production and management practices; and
• establish agri-environmental standards that support the common goals for environment (Agriculture and Agri-Food Canada 2003, sec. 24, 26.3).

Top of Page

2.2 Objectives and Expected Results

The goals of NAESI were stated in the Memorandum of Understanding and were also discussed in the Overarching Report (Bowerman et al. 2009). The goals included establishing non-regulatory environmental standards to support the common goals of the Environment Pillar of the Agricultural Policy Framework. The Memorandum of Understanding identified two types of standards:

• performance standards, which identify a "desired level of environmental state to maintain ecosystem health" and specify the quality levels achievable through existing technology and practice; and
• process standards, which identify the specific practices (beneficial management practices) that support the performance standards (Canada 2003).

Figure 1 shows the logic model for the initiative, depicting the results chain leading from activities to outputs to outcomes. Through NAESI, the Government of Canada undertook data collection, benchmarking and research to produce and disseminate information and guidance related to the four themes for agri-environmental standards: water, biodiversity, pesticides and air. The results of this work supported the realization of the following immediate and intermediate outcomes:

• development of methods for ideal and achievable performance standards; 
• effective technology transfer packages and communications support materials for a range of audiences (e.g. technical, policy-makers and stakeholders); and
• increased understanding of the relationships and links between agriculture and the environment through the collection of scientific information.

The Results-Based Management and Accountability Framework/Risk-Based Audit Framework for NAESI noted that the ultimate strategic outcomes for the initiative, as set out in the logic model, were identified for purposes of the Environment Pillar of the Agricultural Policy Framework. They included improved stewardship of the environment by producers and increased confidence that food is being grown safely and in an environmentally responsible way (Environment Canada 2005, sec. 2.6). 

It is important to note that, with its four years of funding, NAESI was only intended to produce standards; therefore, its ultimate outcome was the creation of these standards. The strategic outcomes identified in the logic model clearly will require extensions to the activities and resources made available for NAESI.

Figure 1.  Logic Model (view large image)

Top of Page

2.3 Financial Resources

The Memorandum of Understanding and the Results-Based Management and Accountability Framework/Risk-Based Audit Framework stipulated the transfer of $25 million to Environment Canada over five years, starting on April 1, 2003. The profile of funding for NAESI changed slightly between the initial conception of the program and the late signing of the Memorandum of Understanding in December 2003, which delayed the start of NAESI until April 1, 2004 (Bowerman et al. 2009). The new allocation was based on the recommendation of the ADM Joint Management Committee (see Section 2.5) of May 13, 2004 (Environment Canada 2005, sec. 3.0). Table 1 shows the allocation of funds, broken down by fiscal year and activity area, as discussed in Section 2.

Source: (Bowerman et al. 2009, pp. 185–186) 

As Table 1 shows, funding for activity area 1, which mostly involved planning, was restricted to the first two years of the initiative. The funding amounted to $2.2 million, or 9 percent of the total NAESI budget. Program delivery and regional coordination accounted for an additional $3.3 million between 2004 and 2008, or about 13 percent. The remaining 78 percent of the funding ($19.5 million) was allocated to activity area 2. Table 1 also shows how the $2.3 million originally set to flow to Environment Canada in 2003–2004 was re-allocated to support program activities in 2004–2008. 

Top of Page

2.4 Initiative Reach: Stakeholders and Partners

Numerous partners were associated with NAESI, including Agriculture and Agri-Food Canada, Environment Canada and the Treasury Board of Canada Secretariat, as well as researchers, contractors, universities and other institutions involved in developing the standards and associated guidance material. Researchers from environmental non-governmental organizations and provincial governments involved in parallel projects also collaborated on the completion of certain NAESI standards.

The intended beneficiaries of NAESI included the agriculture and agri-food sector, the Canadian public and global consumers, who collectively represented NAESI’s target population. The agricultural sector was expected to benefit from information that supports improved agricultural practices that increase sustainability, a better reputation for environmental responsibility and increased ability to respond to “green market opportunities,” as well as continued access to international markets (Roberts 2006; Environment Canada 2005, sec. 3.0). In the long term, it was expected that Canadian and global consumers of Canadian agricultural products would benefit from increased confidence that these products were produced in an environmentally responsible way. NAESI was also expected to benefit the Canadian economy by “improving understanding of environmental costs and benefits for agriculture,” as well as the Canadian public by contributing to “a healthier environment, better quality of life, and sustainable management of natural resources for future generations” (Bowerman et al. 2009, sec. 1.5).

Top of Page

2.5 Governance Structure

NAESI’s governance structure consisted of three levels of management committees. Representatives from each department populated the Deputy Ministers (DM) Committee, the Assistant Deputy Ministers (ADM) Joint Management Committee and the Director General (DG) Steering Committee. The Interdepartmental DG Steering Committee was initially chaired by the Executive Director, Habitat Conservation and Protected Areas, Environmental Stewardship Branch, Environment Canada, and then by the Director General, Public and Resources Sectors Directorate, Environmental Stewardship Branch, Environment Canada. This committee reported to the ADM Joint Management Committee, which, in turn, reported to the DM Committee. The Treasury Board of Canada Secretariat was an ex-officio member on the DG Steering Committee and the ADM Joint Management Committee (Bowerman et al. 2009, p. 6). Four thematic teams, each headed by a theme lead, reported to the NAESI Secretariat, which, in turn, regularly communicated with Agriculture and Agri-Food Canada. NAESI’s governance structure is illustrated in Figure 2. See “2.6 Roles and Responsibilities,” below, for information on the roles and responsibilities of each committee.

Figure 2. NAESI’s governance structure 

Source: Bowerman et al. 2009 

Top of Page

2.6 Roles and Responsibilities

The roles and responsibilities of Environment Canada and Agriculture and Agri-Food Canada included those related to the management of the Memorandum of Understanding (MOU) and those related to the delivery of NAESI. 

Agriculture and Agri-Food Canada reviewed and approved work plans, progress reports and reporting on the implementation of the MOU, in addition to providing funding for NAESI. This department also provided scientific and technical advice and information for other of its agri-environmental initiatives and offered policy advice when the objectives of the MOU changed. During the implementation of NAESI, Agriculture and Agri-Food Canada’s responsibilities included providing advice and guidance, coordinating committee participation, being involved in policy discussions related to standards development, facilitating stakeholder participation, and contributing of agricultural knowledge and expertise. 

Environment Canada led the five-year strategic planning, prepared quarterly progress reports and disseminated information on changes to the objectives of the MOU. Further, it managed the development and execution of annual work plans and oversaw expenditures. Environment Canada also identified priority areas for the development of agri-environmental standards and produced standards in these areas. Finally, this department drafted a series of annual reports and the final synthesis reports discussing the suite of performance standards, and provided recommendations for the use of the standards in the development of process standards by Agriculture and Agri-Food Canada. To carry out these responsibilities, Environment Canada consulted with stakeholders to assess priorities for standards development, and coordinated workshops and other communication activities to discuss and inform the framework for managing agri-environmental standards.

The ADM Joint Management Committee reported to the DM Committee semi-annually and approved annual costs and work plans, reviewed quarterly results and cash flow forecasts and expenditures, approved small changes to the MOU, and recommended changes to the objectives of the MOU to the DG Committee. The DM Committee met annually to discuss the performance of NAESI, recommend to the Treasury Board of Canada Secretariat the amount of funding to be released, approve re-allocations of funds between fiscal years or departments, resolve disputes relating to the implementation of the MOU, and receive approval from the Minister of changes to the MOU’s objectives. The NAESI Secretariat, based in Environment Canada, was responsible for day-to-day management of the activities under the four themes, specifically coordinating annual work plans and budget allotments, and communications activities (see Appendix 4). Each of these entities, as well as the Treasury Board of Canada Secretariat, reviewed the Results-Based Management Accountability Framework/Risk-Based Audit Framework, including the evaluation matrix and the performance measures and indicators.

Top of Page

2.7 Performance Reporting

The following was the reporting strategy for NAESI:

• Environment Canada provided semi-annual reports on the progress of the initiative, using a standard template.
• Environment Canada reported internally on its performance in regards to NAESI in the annual Departmental Performance Report as a mechanism for demonstrating accomplishments.
• Agriculture and Agri-Food Canada used the information Environment Canada provided on this initiative to report on accomplishments in the context of the Agricultural Policy Framework.
• Environment Canada made the final report on the NAESI evaluation available as a public document.

Top of Page

2.8 Initiative Outputs and Outcomes

In addition to the annual Technical Series consisting of a total of 247 reports over four years, NAESI culminated in the development of 16 synthesis reports—15 peer-reviewed technical reports and the Overarching Report. In total, 98 standards were developed and recommended across the four thematic areas. This sub-section discusses the three classes of standards, presents the scale of the standards, and reviews the thematic areas and the associated standards.

Classes of NAESI Standards

NAESI’s Results-Based Management and Accountability Framework/Risk-Based Audit Framework outlined two types of non-regulatory standards considered important for environmental performance: performance standards and process standards. Performance standards are “intended to identify the degree of desired environmental performance,” while process standards “recommend the methodology of production to achieve the performance standards.”⁴ Two types of performance standards were developed: ideal performance standards, which specify “the desired level of environmental state needed to maintain ecosystem health,” and achievable performance standards, which specify “the level of environmental quality that can be achieved using recommended, best available processes and technologies.”⁵ The ideal and achievable states of the environment, as dictated by the standards, need not be equal and may be expected to vary in many circumstances.

For example, Figure 3 depicts a situation that is common in many agricultural contexts. The figure shows that as the magnitude of an agricultural stressor increases, the impact on the environment also increases. In the “current situation,” the magnitude of the stressor is large, and, so, the corresponding environmental impact is high. The desired environmental state, as represented by ideal performance standards, would dictate much less of the stressor, resulting in a much lower impact. However, with available beneficial management practices, it is only feasible to lower but not eliminate the environmental impact.

Figure 3. One potential relationship between ideal and achievable performance standards

Source: Bowerman et al. 2009, sec. 1.6

The Overarching Report noted that “employing different methodologies to develop the standards […] coupled with data availability issues and time constraints led the recommended standards developed through NAESI to vary in format” (Bowerman et al. 2009, sec 1.61). The work undertaken through NAESI, therefore, led to the development of two formats of ideal performance standards, as well as to the development of achievable performance standards. The former were numerical or descriptive benchmarks, and methodological support and guidance relating to standards development. Numerical benchmarks quantified desired environmental conditions, while methodological support and guidance consisted of tools to support the development of standards. The term provisional was used to qualify ideal performance standards of either format developed in situations in which the science was limited and additional basic and applied research would be required before final standards could be identified nationally or regionally (Roberts 2009). 

Scale of NAESI Standards

NAESI standards were developed for a range of geographic scales, as Figure 4 shows. Regions smaller than 1 km² in area were classified as farm or plot-level, regions 10–50 km² were designated as sub-watersheds, and regions with areas of hundreds or thousands of km² were referred to as watersheds/ecoregions or ecozones, respectively. As illustrated, ammonia/particulate matter standards (developed under the air theme) applied to ecozones and watersheds/ecoregions. Standards developed under the water and pesticides themes were applicable at smaller scales (watershed/ecoregions and sub-watershed). Biodiversity standards were most versatile, being applicable to the broadest range of scales. The Overarching Report pointed out that, with the exception of some biodiversity standards, “NAESI standards are not developed for application at the level of the individual farm.” Rather, “they have been designed with the intention for application within environmental boundaries (such as watersheds or ecoregions) to provide a landscape approach to environmental boundaries rather than political or land-ownership boundaries” (Bowerman et al. 2009).

Figure 4. Scale of NAESI standards 

Source: Canada, 2007c, p. 3

NAESI Thematic Areas

NAESI developed standards in four thematic areas: water, biodiversity, pesticides and air. The goals for each theme were as follows:

• Water: “Develop quantitative or qualitative environmental performance standards that provide benchmarks for nutrients, sediments and microbial pathogens in surface waters, as well as unsustainable water withdrawals for agricultural activities” (Bowerman et al. 2009, p. 20).
• Biodiversity: “Deliver a suite of measurable standards representing levels of habitat quantity and quality necessary to support biodiversity conservation, applicable to key combinations of agricultural production types and landscapes across Canada” (Bowerman et al. 2009, p. 18).
• Pesticides: Produce a suite of standards that may “inform and guide future efforts in the reduction of environmental risks posed by the use of pesticides in agriculture” (Bowerman et al. 2009, p. 19).
• Air: “Produce a suite of regional air quality standards that would provide benchmarks against which the impacts of agricultural operations on Canadian air quality can be assessed” (Bowerman et al. 2009, p. 17).

Source: Roberts 2006; Bowerman et al. 2009, sec. 2.0

Key priority areas under the water theme were nutrients, pathogens and instream flow needs. For the biodiversity, pesticides and air thematic areas, key priorities were habitat conservation, priority pesticides in water, and ammonia as a precursor to particulate matter, respectively. Additional subjects were initially considered for inclusion as priorities but ended up not being part of NAESI’s final products. These subjects included water conservation, pesticides in the air, and odour, under the water, pesticides and air themes, respectively. The Overarching Report provided detailed explanations of the factors leading to the exclusion of these topics (Bowerman et al. 2009, sec. 4.0). 

Research undertaken through NAESI led to the development of 98 standards across the four thematic areas. Tables 6-1 through 6-4 in Appendix 6 list the number of standards for each thematic area, each according to whether the standard was classified as an achievable performance standard, a numerical descriptive benchmark, or a methodological or guidance-based standard.

Top of Page

3.0 Evaluation Design

3.1 Purpose and Scope of Evaluation

Since NAESI has ended and is not being considered for renewal, a small-scale evaluation of limited scope was conducted, with a focus on identifying lessons learned rather than recommendations for improvement. Lessons learned may serve to inform the design of future initiatives in the agri-environmental sector.

The following issues were examined in this evaluation:

• Relevance: Was the initiative consistent with departmental and government-wide priorities?
• Design and delivery: Was the initiative designed and delivered in the best possible way?
• Success: Was the initiative effective in achieving its intended outcomes and meeting its objectives?
• Cost-effectiveness: Were the most appropriate, cost-effective and efficient means used to achieve objectives, relative to alternative design and delivery approaches?

A matrix of the evaluation questions, indicators and data sources that guided the evaluation appears in Appendix 1.

The evaluation focused primarily on the five-year timeframe of NAESI (2003–2004 to 2007–2008) but also encompassed some activities in 2008–2009, such as the final NAESI stakeholder workshop, which was held in January 2009. 

Top of Page

3.2 Evaluation Approach and Methodology

The methodology relied on two qualitative lines of evidence: interviews and document review.

Interviews with Stakeholders

A total of 18 interviews were completed with representatives from Environment Canada, Agriculture and Agri-Food Canada, other federal government departments (such as Health Canada’s Pest Management Regulatory Agency), provincial governments and other interested parties, such as environmental non-governmental organizations and industry. Of the 18 interviews, 10 were conducted with federal representatives, 3 with provincial representatives and 5 with representatives of environmental non-governmental organizations and industry. The interviews followed a guide that reflected the evaluation matrix (see Appendix 1). Separate guides were created for the interviews with representatives of the federal government, provincial governments, and environmental non-governmental organizations and industry (see Appendix 2). Interviewees received a copy of the guide prior to the interviews, which were conducted by telephone. Audiotaping, with the interviewee’s permission, ensured all important details were captured in the written notes.

Document Review

A review of relevant documents and literature provided background to prepare a profile of the initiative and respond to specific evaluation questions. The profile is a synopsis of NAESI, including program objectives, the roles and responsibilities of Environment Canada, Agriculture and Agri-Food Canada and other parties, and expected outputs and outcomes. Environment Canada provided most of the documents for review, including the Memorandum of Understanding, the Results-Based Management and Accountability Framework/Risk-Based Audit Framework for NAESI, the framework agreement for Growing Forward, the technical and synthesis reports (including the Overarching Report), and the proceedings of the 2006 and 2009 stakeholder and 2008 technical workshops. Additional documents, such as the Report on Plans and Priorities for each of the lead departments, the 2008 federal budget and the 2007 Speech from the Throne, were found online.

Top of Page

3.3 Limitations of the Evaluation 

The evaluation relied on qualitative evidence drawn from the 18 interviews and the document review. The number and type of stakeholders that could conceivably be affected by the standards and guidelines considerably exceeded the scope of the interviews conducted. Consequently, it is possible that some viewpoints may not have been captured or, more likely, certain perspectives were not emphasized sufficiently. At the same time, the diversity of interviewees was sufficient to offer some confidence that all essential perspectives were recorded. 

The evaluators did not assess the scientific validity of the standards and other technical documentation. The existence of considerable published information and the opinion of those not directly involved in their production support a conclusion that the standards do represent a technically valid system that can guide agri-environmental programming. Of course, the final validation of this will be whether NAESI outputs become the reference for future agri-environmental programming in Canada. Currently, it is too soon to tell.

The final limitation is that this evaluation did not assess the realization of long-term strategic outcomes because NAESI did not involve developing an implementation plan, and the degree to which the standards will be implemented is unclear.

Top of Page

3.4 Evaluation Ratings 

For each evaluation question, a rating was assigned to denote the level of progress made in achieving intended results and to give the reader with an overall view of the performance of NAESI. The ratings are based on a judgment of whether the findings indicate the following:

• that the intended outcomes were achieved;
• that some progress was made toward the intended outcomes; or
• that little progress to date  was made toward the intended outcomes, and it is too early to see the full results.

In situations in which there was compelling subjective evidence that the initiative did well with respect to a given outcome but a complete assessment could not be done due to a lack of performance data, a “~” is placed in front of the rating.

The ratings for all evaluation questions are summarized in Appendix 8.

Top of Page

4.0 FINDINGS

4.1 Relevance

Overall Findings

Consistency with Departmental and Government Priorities

Alignment with Environment Canada Priorities

An assessment of the plans and priorities of the two lead departments provides insight into the extent to which NAESI aligned with these departments’ priorities. Environment Canada’s 2007 Report on Plans and Priorities describes the Department’s three strategic outcomes: the restoration, conservation and enhancement of Canada’s natural capital, risk reduction and contribution to the well-being of Canadians through weather and environmental predictions and services, and protection of Canadians and their environment from the effects of pollution and waste (Environment Canada 2007). NAESI also appears to have aligned with a number of the program areas associated with these strategic outcomes. For example, strategic outcome 1 (Canada’s natural capital is restored, conserved and enhanced) is linked to the following program areas:

1. Biodiversity is conserved and protected.
2. Water is clean, safe and secure.
3. Canadians adopt approaches that ensure the sustainable use and management of natural capital and working landscapes (Environment Canada 2007, p. 18).

Two of the common goals for the environment agreed upon under the Agricultural Policy Framework and incorporated into the goals of NAESI were to “reduce risks and provide benefits to the health and supply of water” and to “ensure compatibility between biodiversity and wildlife” (Agriculture and Agri-Food Canada 2003, sec. 24.1.1, 24.4.4). These aligned well with the first of the two program areas under strategic outcome 1. Moreover, NAESI’s goal of “[evaluating] standards attainable by environmentally-beneficial agricultural production and management practices” would appear to have aligned with the adoption of approaches that ensure the sustainable use of natural capital, the third program area under strategic outcome 1 (Canada, 2003, sec. 1.0). Given the above, it appears NAESI aligned with Environment Canada’s plans and priorities.

Interviewees reported that NAESI aligned well with Environment Canada’s mandate to preserve, restore and enhance Canada’s natural capital. They also said it supported the Department’s focus on air and water quality, and climate change. While climate change was not a NAESI priority, interviewees noted that it was an overarching issue that was considered in all of the work under the initiative. 

Alignment with Agriculture and Agri-Food Canada Priorities

Agriculture and Agri-Food Canada’s 2007–2008 Report on Plans and Priorities outlines three strategic outcomes guiding the activities of the Department: security of the food system, health of the environment and innovation for growth. The second of these was relevant to NAESI and encompassed several priority areas, one of which involved “developing the knowledge to improve the environmental performance of the Canadian agricultural system” (Agriculture and Agri-Food Canada n.d., p. 46). The report states the following:

This priority area appears to have aligned with NAESI’s goal of increasing understanding of relationships between agriculture and the environment (see “2.2 Objectives and Expected Results,” above); therefore, the objectives of NAESI appear to have aligned, in part, with the plans and priorities of Agriculture and Agri-Food Canada.

Alignment with Growing Forward

Interviewees agreed that when it was established NAESI aligned with Agriculture and Agri-Food Canada’s priorities, given that it formed part of the Agricultural Policy Framework. However, this framework has been replaced with Growing Forward, which is a federal/provincial/territorial framework agreement to guide government action in the area of agricultural policy from 2008 to 2012. 

Presenters from Agriculture and Agri-Food Canada at the 2009 Final Stakeholder Workshop (Carona Designs Inc. 2009) noted that NAESI standards could inform policy making with specific reference to the four elements of the Growing Forward policy suite (discover, develop, deliver and determine). This drew attention to the link between NAESI and the Growing Forward policy outcome of developing an agricultural sector that contributes to society’s priorities, including “the capacity to meet the need for strong environmental stewardship and to deliver ecological goods and services” (Agriculture and Agri-Food Canada 2007, sec. 5.2). The evidence suggests that NAESI is consistent with Growing Forward. 

Nonetheless, many interviewees were uncertain whether and how NAESI would be integrated into this policy framework. While agreeing that Growing Forward continues to include the environment as a priority, interviewees said that increased emphasis will be placed on economic sustainability and profitability. They also noted that climate change and water are Growing Forward’s main priorities, with a few interviewees suggesting that biodiversity has become less of a priority.

Alignment with Government-Wide Priorities

Finally, in the 2007 Speech from the Throne, the Governor General identified the protection of the environment as a priority for the Government of Canada. However, the speech did not refer to NAESI or the environmental impact of agriculture, focussing instead on related concerns, such as greenhouse gas emissions and the consequent impact on climate change, as well as air pollution, environmental legislation and enforcement.

Top of Page

4.2 Design and Delivery

Overall Findings

Design

As noted, the formal purpose of NAESI was to develop science-based agri-environmental standards, not including formulating a plan to implement the standards; however, several interviewees said that the intent of NAESI was unclear and that more forethought about how the standards would be used was required (see “4.3 Success,” below). These comments indicate that NAESI did not include an explicit implementation or adoption plan in its design or resource allocation.

Many interviewees also said that the terminology used to describe the outputs created ambiguity about how outcomes would be realized. They explained that the name of the initiative and the terminology in the Memorandum of Understanding suggested that NAESI was intended to lead to the development of agri-environmental standards and that the standards would, in turn, be used to establish regulations for the agricultural sector. At the same time, however, interviewees said that they understood that the intent of NAESI was not to establish standards for regulatory purposes but rather to establish non-regulatory standards, guidelines or performance targets. Several interviewees noted that the softer language used to describe aspects of the initiative (e.g. the word guidelines) may have compromised some stakeholders’ support for the initiative. 

Most interviewees said that the themes selected for inclusion in NAESI were appropriate. They explained that the lead departments worked collaboratively to identify and select a set of themes that would meet their respective needs. Nonetheless, a few interviewees questioned why pesticides was chosen as a theme, noting that pesticides are a stressor that can cause an adverse impact on the environment, in contrast to water, biodiversity and air, which focus on receptors that respond to environmental stressors. Some interviewees suggested that soils should have been included as a theme. However, they noted that Agriculture and Agri-Food Canada had the internal knowledge and expertise required to pursue this topic independently; a few interviewees reported that they had not seen any evidence this department had undertaken this work. Other suggestions for modifications to the themes were to include biodiversity and climate change as overarching themes and to integrate pesticides within the other three themes.

Many interviewees said that a key feature of NAESI’s design was the partnership between the two lead departments. The interviewees reported that the collaboration between the departments was integral to the development of agri-environmental standards, since each brought specific expertise to the table. In addition to fostering inter-departmental relationships, NAESI also created the opportunity for scientists from various sections of Environment Canada to work together. 

Some interviewees suggested that NAESI should have involved a wider range of partners and collaborators, such as provincial governments, and industry and producer associations. However, respondents representing industry associations noted that it is difficult for them to know when to get involved in government programs. Some of the industry associations said that they lack the technical knowledge required to contribute to an initiative such as NAESI. For example, the representative of one producer association said, “We rely on [Agriculture and Agri-Food Canada] to tell us if it is good science.”

Implementation Success

Interviewees who could comment on the implementation of NAESI indicated that it was delivered on time and within budget. However, several noted that the time frame within which the work was to be completed limited the capacity of the initiative to deliver on its strategic outcomes. 

Several interviewees said that the first year of the initiative was spent identifying the themes and setting priorities. Some interviewees suggested that this process could have been completed more quickly. They also noted that this planning phase effectively reduced the amount of time available for the research, standards development and reporting.

While there was the intent to integrate some of the themes, interviewees said that there was limited opportunity to accomplish this. One reason NAESI was unable to thoroughly integrate the themes was that the majority of the time available was spent conducting scientific research and developing the standards. Interviewees noted that each of the standards had to be developed independently prior to considering how they could be integrated. Nonetheless, efforts were made by those working on each theme to keep those working on other themes apprised of their direction and progress. Another reason the themes were not well integrated was that the results of the scientific research conducted were required to inform decisions about how the themes could be integrated. Interviewees reported that discussions about how to integrate the themes are underway; however, they said that without additional funding, this integration might not occur. 

Some interviewees mentioned that integration of the themes is critical to understanding how the standards interact and what potential exists for adverse effects within or across themes. Some interviewees said that preliminary attempts to integrate some of the themes (pesticides and water, and water and biodiversity) were made.

Despite these challenges, interviewees said NAESI resulted in good science and led to the development of some standards. Many noted that it was simply not possible for standards to be developed for each theme and priority within the time available. 

Governance and Communication Processes

Most interviewees agreed that the governance structure for the initiative was effective. The Memorandum of Understanding between the two lead departments and the Treasury Board of Canada Secretariat clearly defined the roles and responsibilities of each department. Environment Canada was responsible for leading each theme, including developing the required work plans, budgets and reports, and conducting the majority of the science. Agriculture and Agri-Food Canada had an advisory role and worked with Environment Canada to identify the thematic areas, provided suggestions for the direction of projects, and reviewed deliverables and reports. 

Some interviewees suggested that there should have been more of an equal partnership between the departments, with them co-leading each of the themes. However, others thought this might be problematic given the departments’ differing mandates and priorities. A few interviewees noted that the theme leads should not have been asked to act as managers for the initiative, since conflicts of interest could arise given the leads’ vested interest in the research. Some interviewees said that inequalities in the partnership negatively influenced departmental commitment to the initiative; some interviewees mentioned that they had observed tension between representatives of the two departments. 

Interviewees reported that one of NAESI’s strengths was the designated program manager, who served as a single point of contact. They noted that the program manager was integral to ensuring effective communication between the lead departments, which occurred, for the most part. 

Top of Page

4.3 Success

Overall Findings

Immediate and Intermediate Outcomes

Note: The “~” symbol refers to the fact that although there was compelling subjective evidence that the initiative did well with respect to this evaluation question, a complete assessment could not be done due to a lack of performance data.

Interviewees reported that one of NAESI’s strengths was its contribution to the federal government’s understanding of the relationships and links between agriculture and the environment. Interviewees said that although Agriculture and Agri-Food Canada already had a strong awareness of the connections between agriculture and the environment, NAESI expanded upon this knowledge base. It also increased this department’s awareness of Environment Canada’s expertise. In addition, NAESI raised Environment Canada’s awareness of agricultural production practices, particularly in the context of working landscapes and business decisions. It also helped Environment Canada understand some of the agri-environmental issues that its counterpart department and the agricultural sector face daily. 

Some interviewees emphasized that NAESI did not increase awareness of issues but rather added to the knowledge of the interactions between agriculture and the environment. The interviewees explained that prior awareness of the issues that needed to be addressed contributed to the identification of the themes that were included in NAESI.

The evaluation findings indicate that this outcome was achieved. According to data provided by the NAESI Secretariat at Environment Canada, 17 technology transfer packages were delivered in various forms, including five National Agri-Environmental Coordinating Committee meetings, two stakeholder meetings, four annual technical series CDs and one final synthesis CD, one brochure and four newsletters. In addition, approximately 75 technical presentations were delivered over the life of the program to a range of stakeholders (e.g. provinces, researchers, academia, conservation authorities, industry and participants at national and international conferences).

Appendix 4 sets out NAESI’s Communications Strategy. A communications specialist ensured that NAESI materials and reports were clear and consistent in tone. The breadth and depth of NAESI communication materials contributed to the transparency of the initiative. Most interviewees said that NAESI did a good job of communicating with stakeholders, but some suggested that NAESI could have engaged stakeholders earlier in the process.

Data from the NAESI Secretariat indicate that 36 methods for standards were developed. These include 21 methods developed as of March 2007 and 15 methods for the final suite of standards (one for the air theme, two for biodiversity, six for pesticides and six for water).

Interviewees noted that solid scientific research was conducted through NAESI. Some of the work expanded on existing information and other work was new research. Interviewees said NAESI generated the scientific research needed to develop standards and lay the foundation for the development of other standards. 

Interviewees explained that ideal performance standards represent the best state of the world, while achievable performance standards represent what can be achieved given the current state of the environment. They noted that the development of the standards was science-driven and that standards simply could not be developed without a scientific foundation. Interviewees reported one of the strengths of NAESI was that all of the research was peer-reviewed.

Ultimate Outcomes

Appendix 5 summarizes the contents of four NAESI newsletters issued between 2006 and 2008, which offered a detailed account of NAESI’s progress in developing the standards. As discussed earlier, 98 standards were developed across the four thematic areas. Table 3 summarizes these standards, based on the information in Appendix 6.

Source: Bowerman et al. 2009, sec. 3.0; Roberts 2009

As set out in Appendix 6, the water, biodiversity, pesticides and air themes produced 16, 17, 64, and 1 standard, respectively. NAESI Secretariat data indicate that the 64 standards developed in the pesticides thematic area represent 100 percent coverage of the 20 priority pesticides identified. In the process of developing the standards, seven semi-annual status updates were provided for the four themes (in March and November 2005, March and November 2006, March and November 2007, and March 2008).

Of the 98 standards, 69 were numerical or descriptive benchmarks, while 13 were methodological or guidance-based. These two types of standards are both categorized as ideal performance standards and account for 82 of the 98 standards. The remaining 16 standards were achievable performance standards (see Tables 6-1 to 6-4 in Appendix 6.) Based on this information, it is apparent that NAESI developed standards in the required themes and sub-themes within the terms of the Memorandum of Understanding.

Interviewees reported that the extent to which NAESI developed standards varied across themes. They explained that this was because the initiative’s ability to establish standards was directly related to the maturity and completeness of the information available upon which standards could be based. In areas in which there was a solid foundation of existing information and knowledge, NAESI was able to articulate standards. When there were gaps in information, NAESI was only able to conduct some of the science needed to set the standards. In these cases, additional research is needed before the standards can be fully defined. Interviewees said that the initiative was the most successful in developing standards under the pesticides theme, with the most work remaining to be done in the areas of water availability, instream flow needs and biodiversity. Interviewees also noted that although odour was originally included under the air theme, it was removed because the scientific foundation was underdeveloped.

Usefulness of Standards

*Although early progress was made in identifying potential uses for some of the NAESI standards, it is not clear whether and how many of the standards will be implemented.

Some interviewees described NAESI as a science program that had the potential to lead to standards development in some areas. They indicated that the timeframe for the initiative was too short to accommodate the completion of the scientific research as well as the implementation of the standards. 

Interviewees had varying opinions on how the standards could be implemented and said that the standards are at various stages of readiness for implementation. Most interviewees said that the standards under the themes of pesticides and water had the most potential for implementation. For example, some of the pesticide standards have been circulated to the Canadian Council of Ministers of the Environment for consideration as Canadian Water Quality Guidelines.

A few interviewees stated that the standards were initially being developed for use in farm certification. However, that concept was abandoned and no clear options for how else the standards could be used were developed. Nonetheless, those involved with NAESI proceeded under the impression that Agriculture and Agri-Food Canada would use the results in its programs. Interviewees said that they do not have a clear understanding of this department’s intentions in this regard and said that a solid implementation plan was not established at the onset of the initiative.

Some interviewees were uncertain how the information and standards generated through NAESI could be utilized, indicating that the work completed through this program is premature and that additional research is needed before the standards will be useful. These interviewees fear that if NAESI were not continued or built upon, the expenditures made through this initiative would be lost.

Other interviewees noted that the standards serve as benchmarks for environmental performance. They noted that the standards can be used to measure the state of the environment and assess the effectiveness of agri-environmental programs. Some interviewees mentioned that provinces are considering how the standards can be used to develop performance standards and inform their work in areas such as land-use planning, environmental farm planning, and the development of ecological goods and services. 

Several interviewees would like to see NAESI incorporated into the development of on-farm programs to demonstrate the impact of agriculture on environmental degradation and to establish beneficial management practices. The interviewees said that the standards could be used as guidance for environmental mitigation and to identify target areas for environmental improvement. 

Some interviewees said the standards are best suited to serve as tools to inform decision making. In some cases, the standards are national, while others are location-specific. This means that users may need to conduct additional research before they can apply the standards to their situation. 

Stakeholders and beneficiaries did not share a common understanding of how the standards developed through NAESI would be used. For some, the name of the initiative and the terminology used to describe it contributed to this lack of clarity. For them, the stated goals seem to suggest that NAESI would result in the development of standards that would be used to establish regulations for the agricultural sector; however, other stakeholders thought that NAESI would develop non-regulatory standards, guidelines or performance targets. The fact that developing implementation processes was not part of the initiative may have contributed to this ambiguity.

Top of Page

4.4 Cost-Effectiveness

Overall Findings

Interviewees said that it was difficult to judge whether NAESI was good value for money. They indicated that a substantial amount of funding was allocated to this initiative but that it was not clear what role NAESI would play in the realization of the long-term strategic outcomes. Although NAESI resulted in good science and produced numerous outputs, more work is needed before some of the information can be used. In addition, not all of the information generated is national in scope.

Other interviewees said that NAESI was good value for money because it provided standards, generated knowledge and science for other areas, and created a partnership between the two lead departments. Interviewees mentioned that the majority of the funding was used for research; admnistration costs were minimal. NAESI also made attempts to partner with other projects, when possible. For example, there were partnerships with the Pesticide Science Fund, the Watershed Evaluation of Beneficial Management Practices initiative, and regional air monitoring stations. 

Several interviewees indicated that some of the themes were not advanced enough given the amount of funding they were allocated. Nonetheless, there were some areas in which the initiative recognized that it would not be able to make a meaningful contribution. In these cases, the work was dropped and another direction pursued. Some interviewees suggested that the government’s three- to five-year funding cycles are too short for science programs to conduct research and demonstrate impacts. 

Interviewees also said that those involved in the sub-themes worked together closely when possible and took advantage of local resources. For example, under the water theme, scientists worked together on field monitoring and hired local farmers and university students to conduct some of the fieldwork so they could minimize travel costs. Under some themes, staff members were able to leverage money from other federal departments. For example, those involved in the air theme engaged in a data-gathering case with Statistics Canada.

Links to Existing Programs

Interviewees were not aware of other programs that are developing agri-environmental standards. Nonetheless, NAESI had links to some existing programs, such as the National Agri-Environmental Health Analysis and Reporting Program (NAHARP) and the Watershed Evaluation of Beneficial Management Practices (WEBs). 

Interviewees explained that NAHARP is developing a suite of standards that could be used to assess the current state of the environment, against which future programs for environmental sustainability can be benchmarked. Interviewees noted the design of NAHARP and NAESI are similar. Like NAESI before it, NAHARP is utilizing a collaborative approach, addressing a range of issues and involving numerous scientists. In addition, both programs are science-based. 

WEBs is measuring the economic and water quality impacts of selected beneficial management practices. Some interviewees noted that some of the projects proposed for NAESI were too closely related to WEBs and, therefore, not funded. However, there was some overlap related to work that explored how implementing a beneficial management practice would impact various standards.

Other programs that NAESI had links to are the Building Public Confidence in Pesticide Regulation and Improving Access to Pest Management Initiative of the Pest Management Regulatory Agency, the Canadian Council of Ministers of the Environment’s Canadian Environmental Quality Guidelines and Water Quality Guidelines for the Protection of Aquatic Life. 

Alternative Approaches

Some interviewees found it difficult to comment on whether NAESI was the best approach to developing agri-environmental standards because there are no clear impacts of the standards to date and it is uncertain how the standards will be used in the future. However, many noted that NAESI was a reasonable approach, and only a few had suggestions for alternative approaches. Some interviewees suggested that NAESI was a science program and, therefore, it should not have been expected to lead to the implementation of the standards it developed. A few interviewees said that it would be important to consider the social factors and economics associated with the standards. 

Top of Page

5.0 Conclusions

NAESI featured collaboration between Environment Canada and Agriculture and Agri-Food Canada on the development of agri-environmental standards. The partnership and relationship created between these departments is seen as one of the main strengths of NAESI. By working together on this initiative, the departments gained an awareness and understanding of the knowledge and expertise of each. 

Although NAESI successfully contributed to the scientific knowledge base of the interactions between agriculture and the environment, it was not able to develop a complete set of national standards for each theme, simply because the science is not sufficiently advanced to support a quantitative benchmark. NAESI did not explicitly include implementation processes or resources, which has created uncertainty about the realization of the long-term strategic outcomes.

Top of Page

5.1 Main Findings

A key finding emerging from this evaluation is the concern of all interviewees about how the standards developed will contribute to the initiative’s expected strategic outcomes. NAESI was a science program designed to establish standards; however, for many stakeholders it remains unclear how the standards will be implemented and utilized. While there were several suggestions for their role—for example, to serve as a baseline for tracking environmental performance, to inform decision making and to form a foundation for future research—interviewees indicated that an implementation strategy for the initiative had not been defined. For example, NAESI appears not to have found expression in Growing Forward, the successor to the Agricultural Policy Framework.

Relevance

The evaluation finds that NAESI aligned with Environment Canada’s strategic outcomes and program areas. It also was part of the Environment Pillar of Agriculture and Agri-Food Canada’s Agricultural Policy Framework and logically is also consistent with Growing Forward, that framework’s successor. At this time, however, it is unclear how NAESI will be integrated into Growing Forward.

Design and Delivery

Stakeholders all said that the four thematic areas were appropriate for agri-environmental programming. The partnership between the two lead departments was also seen as reflecting good design practice, as was the practice of including stakeholders such as industry groups. A few stakeholders said that wider consultation would have been beneficial. Support for the governance structure and the appointment of a single designated program manager came from all interviewees, who saw these as effective delivery processes.  

Success

NAESI increased scientific understanding of the relationship between agriculture and the environment and developed a total of 98 standards. While some variation exists in the number and completeness of the standards developed across the four thematic areas, and work remains to be completed in some areas, interviewees said that a significant body of research has been synthesized to support agri-environmental policy and programming.

For some interviewees the fact that NAESI was intended to support the development of non-regulatory standards, guidelines and performance targets created ambiguity, since the strategic outcomes clearly require implementation and adoption within programs and regulations. Without an implementation plan, however, it is unclear how the NAESI standards will result in the long-term strategic outcomes.

Cost-Effectiveness

The administrative costs of NAESI were low, and the initiative did not duplicate existing programming in Canada. NAESI linked to important federal agri-environmental initiatives, and most interviewees said that the initiative generated state-of-the art scientific information; however, it is difficult to judge the value of NAESI, since it is unclear whether and how the standards developed will be implemented.

Top of Page

5.2 Key Lessons Learned 

Since NAESI has ended, the focus of this evaluation was on lessons learned as opposed to recommendations for improvement. Lessons learned from the experience with NAESI may serve to inform the design of future similar initiatives.

The following are the lessons learned from NAESI.

1. Clearly stating the intent and objectives of the initiative at its outset, especially with respect to implementation, increases partner and stakeholder support and buy-in. This also involves ensuring all stakeholders understand the language used in memoranda of understanding and communication materials. Had this happened for this initiative, work on the incomplete standards may have increased and stakeholder implementation of NAESI’s outputs may have been encouraged.
2. A single program manager and point of contact proved instrumental to this horizontal initiative. The manager facilitated interdepartmental communication and coordinated diverse technical initiatives.
3. Maintaining strong relationships and communication with partners is essential not only to the creation of an initiative’s intended outputs but also for the realization of its intended outcomes. For an initiative such as NAESI, the standards developed must be integrated into agri-environmental programming that supports on-farm adoption of beneficial management practices. Agriculture and Agri-Food Canada has primary responsibility for ensuring that NAESI outputs become integrated into the agri-environmental programming under Growing Forward. 
4. The primary objective of NAESI was to develop agri-environmental standards. While NAESI developed national standards, synchronization with existing or planned provincial standards or regulations remains unfinished. Consultation with provincial stakeholders at an earlier stage of the initiative might have enabled NAESI to become integrated with provincial processes. This might also have helped stimulate additional support for the initiative. 
5. Defining an implementation strategy for the outputs is essential for their integration into agri-environmental programming. Although NAESI added to the scientific knowledge base and established standards, it lacked an implementation strategy; therefore, it cannot be said that it has progressed toward its strategic outcomes. In the final analysis, realization of the strategic outcomes will require an implementation plan to promote the use and application of the standards through the agri-environmental programming under Growing Forward.

Top of Page

Appendix 1

Evaluation Issues and Questions

Top of Page

Appendix 2

Interview Guides

EVALUATION OF THE NATIONAL AGRI-ENVIRONMENTAL STANDARDS INITIATIVE

INTERVIEW GUIDE
FEDERAL GOVERNMENT REPRESENTATIVES

Introduction

The Audit and Evaluation Branch of Environment Canada (EC) is conducting an evaluation of the National Agri-Environmental Standards Initiative (NAESI). A brief overview of the program is appended to this guide. The objective of this evaluation is to assess issues related to the program’s relevance, design and delivery, success, and cost-effectiveness. 

EC has hired PRA Inc., a private research firm, to conduct the evaluation, which comprises a document review and key informant interviews. The questions below serve to guide the interview process. If you are unable to answer a specific question or it does not apply to you, please tell the interviewer and we will skip that question. Please note that your responses will not be linked to you in the evaluation report.

1. Please briefly describe your relationship to and/or involvement with NAESI.

• For how long have you had some involvement with the initiative?
• With which themes (air, biodiversity, pesticides, and water)/standards have you been most involved? What were your specific roles and responsibilities?
• What stakeholders (federal or provincial governments, producer organizations, non-profit organizations, etc.) have you worked with as part of this initiative?

Relevance

2. How well does NAESI align with the priorities of the federal government? (EQ1)

• How does it support the current goals of EC?
• How does it support the current goals of Agriculture and Agri-Food Canada (AAFC)?
• How does it align with the Next Generation of the Agricultural Policy Framework: Growing Forward?

Design and Delivery

3. Based on your experience with NAESI, how effective is the design of the initiative? (EQ2)

• Are the themes (air, biodiversity, pesticides, and water) appropriate? Why or why not? What other themes, if any, should NAESI include?
• How well integrated are the four themes? What other opportunities for integration exist?
• What are the pros and cons of developing ideal performance standards (IPS) and achievable performance standards (APS)?
• How do science results support the development of agri-environmental standards?

4. To your knowledge, was NAESI (that is, the initiative as a whole and not the individual standards) implemented as intended? (EQ3)

• Were the implementation steps well designed and executed on time and within budget?
• What factors contributed to the successful implementation of the initiative?
• What factors impeded the successful implementation of the initiative? How were these factors addressed?

5. Now think about the governance structure for NAESI. (EQ4)

• Are roles and responsibilities of EC and AAFC clearly defined, accepted, and understood?
• To what extent have EC and AAFC acted upon their responsibilities?
• To what extent have EC and AAFC shown leadership on NAESI projects?
• What are the strengths and weaknesses of EC and AAFC’s partnership for this initiative? What opportunities exist to strengthen the partnership?

6. Please assess the effectiveness of processes used to manage: (EQ4)

• Internal communications within EC and AAFC
• Communications between EC and AAFC 
• External communications with provincial governments, producer associations, non-profit organizations, etc.

What improvements should be undertaken, if any?

7. From your point of view, what have been the best practices and key lessons learned relating to the design and delivery of NAESI? (EQ2/3)

• What have been the strengths and best practices in the design and delivery of the initiative? What has worked well?
• What have been the challenges or limitations? What has not worked well?
• What improvements would you suggest to the design and delivery of NAESI?

Success

8. Based on your experience, to what extent have each of the following intended immediate and intermediate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ5)

• Increased awareness and understanding of the relationships and linkages between agriculture and the environment
• The incorporation of science results into standards development
• Development of methods for ideal and achievable performance standards (Prompt: Have an appropriate number of standards been developed? How effective are the standards that have been developed?)
• The development and distribution of technology transfer and communications support materials (Prompt: What types of information packages have been created? Who have they been distributed to?)
• Increased awareness and understanding of NAESI results and standards

9. Based on your experience, to what extent have each of the following intended ultimate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ6)

• Development of ideal and achievable standards that are national, transparent, and coherent with existing roles and responsibilities (regulations) across government and non-government sectors for the following themes:

10. In your view, was an appropriate level of effort devoted to developing standards for each of the theme areas (air, biodiversity, pesticides, and water)? For each theme… (EQ6)

1. Was an appropriate suite of standards developed? (Prompt: Do the suites comprise the right number and type [ideal performance standards versus achievable performance standards] of standards?) (EQ2)
2. To what extent does the suite of standards reflect the priorities of the theme?

11. How useful are the set of standards that were developed through NAESI? (EQ7)

1. How can the standards developed be used? 
2. To what extent can the standards be implemented? When can they be implemented? What programs are in place to facilitate their implementation? What additional programs are needed?
3. What synergies across other agri-environmental programs has NAESI facilitated?
4. To what extent have the results from NAESI been leveraged to be useful for other activities (e.g., other regulatory processes)?

Cost Effectiveness and Alternatives

12. Are you aware of any other government programs or initiatives that have similar objectives to those of NAESI? If yes, please specify. (EQ8)

1. How is NAESI similar to these programs or initiatives? How is it different?
2. Are any efforts made to avoid duplication of effort and facilitate the complementarity of NAESI and other government programs or initiatives? If yes, please describe.

13. In your view, has NAESI provided good value for money? (EQ9)

1. What features of NAESI are cost-effective? What features are cost-ineffective?
2. How could the cost-effectiveness of NAESI be improved?
3. In your view, is NAESI the best approach or are there alternative approaches to developing national agri-environmental performance standards that would achieve the same or better results?

14. Do you have any final comments about NAESI?

Thank you for your cooperation.

Overview of the National Agri-Environmental Standards Initiative

The National Agri-Environmental Standards Initiative (NAESI) is a four-year, $25 million program. Its objective is to develop national environmental performance standards for agricultural production. These standards will, either qualitatively or quantitatively, establish the degree of desired environmental quality.

The program’s anticipated immediate and intermediate outcomes include:

• Policy-makers’ use of science-based standards to inform decision-making
• Linkage of standards with other initiatives such as the National Agri-Environmental Health Analysis and Reporting Program (NAHARP) and the Watershed Evaluation of Beneficial Management Practices (WEBs) for the purposes of program monitoring and performance measurement.

EC and AAFC are developing two types of standards: Ideal Performance Standards, which specify the desired level of environmental state needed to maintain ecosystem health, and Achievable Performance Standards, which specify the level of environmental quality that can realistically be achieved using currently available and recommended beneficial management practices.

Standards are being developed in four areas: 

• Water (aquatic ecosystem health): nutrients, sediments, and instream flow needs
• Biodiversity (aquatic and terrestrial ecosystem health): quality and quantity of habitat to support key components of biodiversity (survival, reproduction) in aquatic and terrestrial ecosystems
• Pesticides (aquatic and terrestrial ecosystem health): reduce the risk of pesticides
• Air (human health): farm-level ammonia emissions. 

EVALUATION OF THE NATIONAL AGRI-ENVIRONMENTAL STANDARDS INITIATIVE

INTERVIEW GUIDE
PROVINCIAL GOVERNMENT REPRESENTATIVES

Introduction

The Audit and Evaluation Branch of Environment Canada (EC) is conducting an evaluation of the National Agri-Environmental Standards Initiative (NAESI). A brief overview of the program is appended to this guide. The objective of this evaluation is to assess issues related to the program’s relevance, design and delivery, success, and cost-effectiveness. 

EC has hired PRA Inc., a private research firm, to conduct the evaluation, which comprises a document review and key informant interviews. The questions below serve to guide the interview process. If you are unable to answer a specific question or it does not apply to you, please tell the interviewer and we will skip that question. Please note that your responses will not be linked to you in the evaluation report.

1. Please briefly describe your relationship to and/or involvement with NAESI.

1. For how long have you had some involvement with the initiative?
2. With which themes (air, biodiversity, pesticides, and water)/standards have you been most involved? What were your specific roles and responsibilities?
3. What stakeholders (federal or provincial governments, producer organizations, non-profit organizations, etc.) have you worked with as part of this initiative?

Design and Delivery

2. Based on your experience with NAESI, how effective is the design of the initiative? (EQ2)

1. Are the themes (air, biodiversity, pesticides, and water) appropriate? Why or why not? What other themes, if any, should NAESI include?
2. How well integrated are the four themes? What other opportunities for integration exist?
3. What are the pros and cons of developing ideal performance standards (IPS) and achievable performance standards (APS)?
4. How do science results support the development of agri-environmental standards? 

3. To your knowledge, was NAESI (that is, the initiative as a whole and not the individual standards) implemented as intended? (EQ3)

1. Were the implementation steps well designed and executed on time and within budget?
2. What factors contributed to the successful implementation of the initiative?
3. What factors impeded the successful implementation of the initiative? How were these factors addressed?

4. From your point of view, what have been the best practices and key lessons learned relating to the design and delivery of NAESI? (EQ2/3)

1. What have been the strengths and best practices in the design and delivery of the initiative? What has worked well?
2. What have been the challenges or limitations? What has not worked well?
3. What improvements would you suggest to the design and delivery of NAESI?

Success

5. Based on your experience, to what extent have each of the following intended immediate and intermediate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ5)

1. Increased awareness and understanding of the relationships and linkages between agriculture and the environment
2. The incorporation of science results into standards development
3. Development of methods for ideal and achievable performance standards (Prompt: Have an appropriate number of standards been developed? How effective are the standards that have been developed?)
4. The development and distribution of technology transfer and communications support materials (Prompt: What types of information packages have been created? Who have they been distributed to?)
5. Increased awareness and understanding of NAESI results and standards

6. Based on your experience, to what extent have each of the following intended ultimate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ6)

1. Development of ideal and achievable standards that are national, transparent, and coherent with existing roles and responsibilities (regulations) across government and non-government sectors for the following themes:

7. In your view, was an appropriate level of effort devoted to developing standards for each of the theme areas (air, biodiversity, pesticides, and water)? For each theme… (EQ6)

1. Was an appropriate suite of standards developed? (Prompt: Do the suites comprise the right number and type [ideal performance standards versus achievable performance standards] of standards?) (EQ2)
2. To what extent does the suite of standards reflect the priorities of the theme?

8. How useful are the set of standards that were developed through NAESI? (EQ7)

1. How can the standards developed be used? 
2. To what extent can the standards be implemented? When can they be implemented? What programs are in place to facilitate their implementation? What additional programs are needed?
3. What synergies across other agri-environmental programs has NAESI facilitated?
4. To what extent have the results from NAESI been leveraged to be useful for other activities (e.g., other regulatory processes)?

Cost Effectiveness and Alternatives

9. Are you aware of any other government programs or initiatives that have similar objectives to those of NAESI? If yes, please specify. (EQ8)

1. How is NAESI similar to these programs or initiatives? How is it different?
2. Are any efforts made to avoid duplication of effort and facilitate the complementarity of NAESI and other government programs or initiatives? If yes, please describe.

10. In your view, has NAESI provided good value for money? (EQ9)

1. What features of NAESI are cost-effective? What features are cost-ineffective?
2. How could the cost-effectiveness of NAESI be improved?
3. In your view, is NAESI the best approach or are there alternative approaches to developing national agri-environmental performance standards that would achieve the same or better results?

11. Do you have any final comments about NAESI?

Thank you for your cooperation.

Overview of the National Agri-Environmental Standards Initiative

(Repeated from previous guide)

EVALUATION OF THE NATIONAL AGRI-ENVIRONMENTAL STANDARDS INITIATIVE

INTERVIEW GUIDE
PRODUCER ASSOCIATIONS AND NON-PROFIT ORGANIZATIONS

Introduction

The Audit and Evaluation Branch of Environment Canada (EC) is conducting an evaluation of the National Agri-Environmental Standards Initiative (NAESI). A brief overview of the program is appended to this guide. The objective of this evaluation is to assess issues related to the program’s relevance, design and delivery, success, and cost-effectiveness.

EC has hired PRA Inc., a private research firm, to conduct the evaluation, which comprises a document review and key informant interviews. The questions below serve to guide the interview process. If you are unable to answer a specific question or it does not apply to you, please tell the interviewer and we will skip that question. Please note that your responses will not be linked to you in the evaluation report.

1. Please briefly describe your relationship to and/or involvement with NAESI.

1. For how long have you had some involvement with the initiative?
2. With which themes (air, biodiversity, pesticides, and water)/standards have you been most involved? What were your specific roles and responsibilities?
3. What stakeholders (federal or provincial governments, producer organizations, non-profit organizations, etc.) have you worked with as part of this initiative?

Design and Delivery

2. To your knowledge, was NAESI (that is, the initiative as a whole and not the individual standards), implemented as intended? (EQ3)

1. Were the implementation steps well designed and executed on time and within budget?
2. What factors contributed to the successful implementation of the initiative?
3. What factors impeded the successful implementation of the initiative? How were these factors addressed?

3. From your point of view, what have been the best practices and key lessons learned relating to the design and delivery of NAESI? (EQ2/3)

1. What have been the strengths and best practices in the design and delivery of the initiative? What has worked well?
2. What have been the challenges or limitations? What has not worked well?
3. What improvements would you suggest to the design and delivery of NAESI? 

Success

4. Based on your experience, to what extent have each of the following intended immediate and intermediate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ5)

1. Increased awareness and understanding of the relationships and linkages between agriculture and the environment
2. The incorporation of science results into standards development
3. Development of methods for ideal and achievable performance standards (Prompt: Have an appropriate number of standards been developed? How effective are the standards that have been developed?)
4. The development and distribution of technology transfer and communications support materials (Prompt: What types of information packages have been created? Who have they been distributed to?)
5. Increased awareness and understanding of NAESI results and standards

5. Based on your experience, to what extent have each of the following intended ultimate outcomes been achieved as a result of NAESI? Please comment on only those outcomes which are applicable to you and about which you are knowledgeable, providing evidence or concrete examples of impacts where possible. (EQ6)

1. Development of ideal and achievable standards that are national, transparent, and coherent with existing roles and responsibilities (regulations) across government and non-government sectors for the following themes:
   1. Air (particulate matter/ammonia)
   2. Biodiversity (habitat conservation)
   3. Pesticides (high risk, commodity) (Prompt: For what percentage of high priority pesticides have standards been developed?)
   4. Water (nutrients, pathogens, sediment, instream flow needs, quantity/conservation)

6. In your view, was an appropriate level of effort devoted to developing standards for each of the theme areas (air, biodiversity, pesticides, and water)? For each theme… (EQ6)

1. Was an appropriate suite of standards developed? (Prompt: Do the suites comprise the right number and type [ideal performance standards versus achievable performance standards] of standards?) (EQ2)
2. To what extent does the suite of standards reflect the priorities of the theme?

7. How useful are the set of standards that were developed through NAESI? (EQ7)

1. How can the standards developed be used? 
2. To what extent can the standards be implemented? When can they be implemented? What programs are in place to facilitate their implementation? What additional programs are needed?
3. What synergies across other agri-environmental programs has NAESI facilitated?
4. To what extent have the results from NAESI been leveraged to be useful for other activities (e.g., other regulatory processes)?

Cost Effectiveness and Alternatives

8. Are you aware of any other government programs or initiatives that have similar objectives to those of NAESI? If yes, please specify. (EQ8)

1. How is NAESI similar to these programs or initiatives? How is it different?
2. Are any efforts made to avoid duplication of effort and facilitate the complementarity of NAESI and other government programs or initiatives? If yes, please describe. 

9. In your view, has NAESI provided good value for money? (EQ9)

1. What features of NAESI are cost-effective? What features are cost-ineffective?
2. How could the cost-effectiveness of NAESI be improved?
3. In your view, is NAESI the best approach or are there alternative approaches to developing national agri-environmental performance standards that would achieve the same or better results?

10. Do you have any final comments about NAESI?

Thank you for your cooperation.

Overview of the National Agri-Environmental Standards Initiative

(Repeated from previous guide)

Top of Page

Appendix 3

References

Agriculture and Agri-food Canada..2003. Federal-Provincial-Territorial Framework Agreement on Agricultural and Agri-Food Policy for the Twenty-First Century. Available from http://www4.agr.gc.ca/AAFC-AAC/
display-afficher.do?id=1201110596840&lang=eng#s1 (accessed March 2, 2009).

Agriculture and Agri-Food Canada. n.d. 2007–2008 Report on Plans and Priorities. Available from http://www.TBS-sct.gc.ca/RPP/0708/
AAFC-aac/AAFC-aac-eng.pdf (PDF) (accessed March 11, 2009).

Agriculture and Agri-Food Canada. 2007. Growing Forward: A Federal-Provincial-Territorial Framework Agreement on Agriculture, Agri-Food and Agri-Based Products Policy. Available from http://www4.agr.gc.ca/AAFC-AAC/
display-afficher.do?id=1224167497452&lang=eng (accessed March 11, 2009).

Belanger, D. 2008. February 5–6, 2008 Technical Meeting Report. National Agri-Environmental Standards Initiative Technical Series Report No. 4-62. 230 p.

Bowerman, M., L. Maclean, S. Villeneuve, and E. S. Roberts. 2009. National Agri-Environmental Standards Initiative Overarching Report. National Agri-Environmental Standards Initiative Synthesis Report No. 1. Environment Canada. Gatineau, Quebec. 239 p.

Canada. 2003. Memorandum of Understanding: Agricultural Policy Framework Initiatives Development (cited in Bowerman et al., 2009). December 2003.

Canada. 2007a Speech from the Throne. Available from: http://www.ctv.ca/servlet/ArticleNews/
story/CTVNews/20071016/thronespeech_SIDEBARS_071016/20071016/ (accessed March 3, 2009). 

Canada. 2007b. National Agri-Environmental Standards Initiative Brochure. Environment Canada and Agriculture and Agri-Food Canada.

Canada. 2007c. National Agri-Environmental Standards Initiative, Newsletter No. 2. Environment Canada and Agriculture and Agri-Food Canada. July 2007.

Carona Designs Inc. 2009. National Agri-Environmental Standards Initiative (NAESI) Final Stakeholder Workshop. January 7–8, 2009. National Agri-Environmental Standards Initiative. Environment Canada. 134 p.

Ellis, D. 2006. National Agri-Environmental Standards Initiative (NAESI) Stakeholder Consultation Workshop—Proceedings. National Agri-Environmental Standards Initiative Technical Series Report No. 2-71. 81 p.

Environment Canada. 2005. Results-based Management and Accountability Framework (RMAF) and Risk Based Accountability Framework (RBAF) for the National Agri-Environmental Standards Initiative (NAESI). June 1, 2005.

Environment Canada. 2007. 2007-2008 Report on Plans and Priorities. Available from http://www.tbs-sct.gc.ca/rpp/0708/
ec-ec/ec-ec-eng.pdf (PDF) (accessed March 3, 2009)

Environment Canada. 2008. Evaluation Plan for the National Agri-Environmental Standards Initiative (NAESI).

Lefebvre, A. 2006. National Agri-Environmental Standards Initiative (NAESI). Presentation at Environment Chapter Leads Meeting. Ottawa, Ontario. February 8–9, 2006.

Roberts, E. 2006. The Program. Presentation at NAESI Stakeholder Consultation Workshop Toronto, Ontario. March 21, 2006

Roberts, E. 2009. Overview of NAESI. Presentation at final NAESI Stakeholder Workshop Winnipeg, Manitoba. January 7, 2009.

Top of Page

Appendix 4

NAESI Communications Activities

The initiative’s Communications Strategy identified the responsibilities of Environment Canada (EC) and Agriculture and Agri-Food Canada (AAFC) for communicating the purpose, methodologies, results and uses of the standards, and was designed to guide managers, the Secretariat, regional coordinators, and theme and project leads. 

Roles and Responsibilities Related to Communications

Stakeholder roles and responsibilities pertaining to communications activities were specified in the NAESI Communications Strategy, which was an appendix to the Overarching Report, and are outlined in Table 4-1. 

Source: Bowerman et al. 2009, pp. 187–195

NAESI Communications Products

The Communications Strategy outlined current and ongoing federal and external communications activities, as well as tools used to conduct activities, which included stakeholder consultations, the NAESI brochure and website, the annual NAESI Technical Series, and the newsletter. The Strategy described each tool, outlined its objectives and identified its target audience, noting that communications tools originated from the Secretariat and that communications products intended for the public were developed with and approved by the AAFC liaison.

Through the four years, two stakeholder consultation workshops were held. The first, held in Toronto in March 2006, focused on increasing the understanding about NAESI, identifying the uses of the standards among participants, disseminating information on programs into which the standards could link, and developing recommendations to address participant concerns related to communication and standards use. The NAESI brochure and newsletter were created in response to stakeholder feedback from this first workshop. 

The second stakeholder workshop, held in Winnipeg in January 2009, provided an opportunity for EC and AAFC representatives to present the draft standards, discuss limitations and gaps in the science, and identify opportunities to use the standards to inform decision-making in agriculture (Carona Designs Ltd. 2009).

The NAESI brochure (Canada 2007b), introduced in April 2007, provided an overview of NAESI for stakeholders and the public, including partners. It outlined the objectives of NAESI, potential uses of standards, as well as links to other APF programs. The newsletter, introduced in January 2007, provided regular updates to stakeholders regarding the findings and presented examples of how standards could be employed. In total, four issues of the newsletter were published between January 2007 and October 2008. The NAESI Technical Series, intended for the informed public and scientific community, produced 247 reports released in four annual issues between 2005 and 2008. The reports detailed the program’s technical findings, which formed the foundation of the standards. 

The Communications Strategy then outlined the dissemination of NAESI results in particular settings. The Secretariat reviewed the messaging and policy implications in the materials and incorporated any comments from AAFC. In accordance with the Strategy, EC summarized work done in each thematic area in a series of synthesis reports, including discussions of key findings, methodology, sampling locations and draft standards.

The Strategy outlined the peer review process, which was completed by March 2008. This process involved the review of each NAESI technical synthesis report, different from a technical report, by a minimum of three independent researchers, with the process overseen by an independent evaluator. The submission of the NAESI technical synthesis to the peer review process ensured the scientific validity of the standards.  

The value of the expertise that resided within NAESI’s stakeholder groups was recognized by EC and AAFC, and a stakeholder review was carried out in conjunction with the 2009 stakeholder workshop. This review provided involved stakeholder organizations with an opportunity to ask questions and provide technical advice and recommendations to report authors.

Top of Page

Appendix 5

NAESI Timeline

Top of Page

Appendix 6

NAESI Standards by Thematic Area

Water Standards

As shown in Table 6-1, the 16 standards for the water theme were developed across five topics, including nutrients (toxicity and eutrophication), sediments, pathogens, water availability and instream flow needs. Three of the standards were classified as provisional standards, including the numerical or descriptive benchmark for pathogens, and the methodological and guidance-based standards for water availability and instream flow needs.

Biodiversity Standards

As shown in Table 6-2, a total of 17 standards were developed for the biodiversity theme, across six topics: natural areas, riparian ecosystems, forest ecosystems, grasslands ecosystems, wetlands ecosystems and anthropogenic areas. All of the standards were classified as numerical or descriptive benchmarks.

Pesticide Standards

As shown in Table 6-3, 64 standards were developed for the pesticide theme, across six topics: IPS, APS, risk-based standards, mixtures standards, commodity-based standards and meteorological standards.  

Air Standards

As shown in Table 6-4, one standard was developed for the air theme, associated with ammonia as it relates to the formation of airborne particulate matter. 

Top of Page

Appendix 7

NAESI Outputs

Top of Page

Appendix 8

Summary of Findings

Notes

~√ Although there is compelling subjective evidence that the initiative has done well with respect to this evaluation question, a complete assessment cannot be done due to a lack of performance data.

X* Although early progress has been made in identifying potential uses for some of the NAESI standards, it is not clear whether and how many of the standards developed will be used or implemented.

1 - In actual fact, funds were not provided to Enivronment Canada until April 1, 2004. Work commenced on that date as well. See “2.3 Financial Resources,” below, for more information. Back to top

2 - In actual fact, funds were not provided to Environment Canada until April 1, 2004. Work commenced on that date as well. See “2.3 Financial Resources,” below, for additional details. Back to top

3 - This section draws from Environment Canada (2005), Roberts (2006, 2009) and Bowerman et al. (2009). Back to top

4 - Under this framework, Environment Canada is responsible for the performance standards, while Agriculture and Agri-Food Canada is responsible for the process standards. Back to top

5 - While the definition of ideal performance standards is unchanged from the accountability framework in the Overarching Report, the definition for acheiveable performance standards appears to have been slightly revised to read, “environmental conditions that can realistically be achieved using a variety of land management scenarios including currently available and recommended [beneficial management practices]” (Bowerman et al. 2009, sec. 1.6). Back to top