Warning This Web page has been archived on the Web.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the Contact Us page.

Evaluation of the Canadian Environmental Sustainability Indicators (CESI) Initiative

Final Report

February 26, 2009

Table of Contents

Report Clearance Steps

Planning phase completedJune 2008
Report sent for management responseJanuary 2009
Management response receivedJanuary 2009
Report completedJanuary 2009
Report approved by Departmental Evaluation Committee (DEC)February 2009

Acronyms used in the report

CAPMoNCanadian Air and Precipitation Monitoring Network
CCMECanadian Council of Ministers of the Environment
CEPACanadian Environmental Protection Act
CESICanadian Environmental Sustainability Indicators
CISECanadian Information System for the Environment
DGDirector General
ENGOEnvironmental non-governmental organization
ESDIEnvironmental and Sustainable Development Indicators
FSDAFederal Sustainability Development Act
FSDSFederal Sustainability Development Strategy
GDPGross domestic product
IISDInternational Institute for Sustainable Development
IPCCIntergovernmental Panel on Climate Change
ISInternal Services
NAPSNational Air Pollution Surveillance
NGONon-governmental organization
NPRINational Pollutants Release Inventory
NRTEENational Round Table on the Environment and the Economy
OECDOrganization for Economic Co-operation and Development
PM2.5Fine particulate matter
SIIDStrategic Information Integration Directorate
WQIWater Quality Index

Acknowledgments

The Evaluation Project Team, including Karine Kisilenko and Martine Perrault under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights and comments crucial to this evaluation.

Prepared by the Audit and Evaluation Branch of Environment Canada.

EXECUTIVE SUMMARY

Introduction

This report presents the results of the evaluation of the Canadian Environmental Sustainability Indicators (CESI) initiative conducted by Environment Canada’s Audit and Evaluation Branch between March and December 2008.

This evaluation focused on assessing the CESI initiative’s relevance, success, cost effectiveness and design and delivery since its inception in 2005. It relied on a review of documents, including recent external st udies of the initiative’s relevance and effectiveness, and 11 key informant interviews with CESImanagers from Environment Canada, Statistics Canada and Health Canada.

The CESIinitiative is a collaborative effort of Environment Canada, Statistics Canada and Health Canada, with input from Canadian provinces and territories initiated by the Government of Canada to develop and report on a small set of priority environmental indicators, notably air quality, water quality and greenhouse gas emissions. The indicators are intended to help provide Canadians with a better understanding of the relationships that exist among the economy, the environment and human health and well-being with respect to air quality, water quality and greenhouse gas emissions. The indicators are also intended to assist those in government who are responsible for developing policy and measuring performance.

The funding for the initiative includes $45 million for fiscal years 2005–2006 to 2008–2009, divided among Environment Canada ($32.5 million), Statistics Canada ($10.5 million) and Health Canada ($2 million).

Summary of Findings and Conclusions

Overall, evidence shows that the CESIinitiative is relevant to both federal government priorities and various stakeholders’ environmental information needs. While other similar sources of environmental sustainability data exist, the initiative appears to have a unique mandate and to involve collaboration among the key government stakeholders (i.e., Environment Canada, Health Canada, Statistics Canada, and the provinces and territories).

However, while the CESIinitiative has produced its intended outputs and achieved or is on track to achieving most of its outcomes pertaining to increased capacity to produce environmental indicators, no evidence was found that CESIproducts are actually being used, except by Treasury Board in its annual Canada’s Performance reports and by Human Resources and Social Development Canada in its set of human well-being indicators. Attention is needed with respect to achieving intermediate outcomes targeting the use of CESI information by decision makers and the general public, as well as the integration of CESIresults in policy decisions.

In 2007, CESImanagers contracted two external studies aiming, among other objectives, to measure the awareness and perceived usefulness of CESIproducts by CESItarget users, namely, senior level policy makers, experts in accountability reporting, members of the general public with an interest in environmental issues, and representatives from environmental non-governmental organizations (ENGOs) and industry. Results from these studies revealed that CESIproducts were not well known and had design flaws that limited their usefulness.

In order to address these shortcomings, and based on recommendations from these two studies, CESImanagers are currently making enhancements to the CESIwebsite and the 2008 suite of products. These improvements include presenting CESIinformation in more common language, giving access to data at more specific levels (drill-down capacity), providing more trend information, and better integrating socio-economic and health context information.

One recommended change could, however, not be implemented. Several federal government policy makers and a few CESIrepresentatives argued that the use of CESIdata by federal government decision makers and policy makers would increase if it could be used for policy development and accountability reporting purposes. This is, however, seen as being dependent on the development of a federal environmental sustainability strategy and the identification of specific objectives against which the federal government would be measured using CESIindicators. No such objectives currently exist.

In light of the improvements that are currently being made to the initiative, it is too early to draw definitive conclusions on the adequacy of the CESIinitiative to meet its target users’ need. Once these changes are fully implemented, the initiative will need to demonstrate its usefulness by documenting who its target users are and by monitoring the extent to which CESIproducts are known, accessed, used and considered useful by these target users.

The initiative currently lacks a performance measurement system to effectively monitor whether its target users are accessing and using the CESIinformation. The number of responses to the two user surveys conducted by Statistics Canada was too low for the results to be meaningful. Furthermore, evidence shows that there is generally low awareness of CESIproducts among the general public and non-government stakeholders, suggesting that a more effective information dissemination strategy may also be required.

Nevertheless, the initiative is generally implemented as designed and in an efficient manner. It benefits from an appropriate and effective governance structure that reflects the interdepartmental nature of its mandate and activities. CESImanagers have also demonstrated a best results-management practice in producing meeting minutes, records of decisions and a review of CESIimplementation challenges that led to effective solutions being developed.

Recommendations

The following recommendations are based on the findings and conclusions of the evaluation.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESIproducts by its target audiences, the initiative needs to monitor the extent to which CESIproducts are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESIproducts. It is recommended that the CESIDG Steering Committee oversee development and implementation of a performance measurement system for the CESIinitiative. This system should include a database of target users of CESIproducts, a process for annually measuring user reach, satisfaction and use of CESIproducts, and a reporting strategy.

Recommendation 2: Several sources indicated that the target CESIusers have limited awareness of the CESIproducts and website. Given the importance of increasing awareness of the CESIproducts, it is recommended that the CESIDG Steering Committee revisit the CESIcommunications strategy to identify and implement means to increase target CESIusers’ awareness of CESIproducts.

Recommendation 3: In light of the significant resource investments made in the CESIinitiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESIproducts. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESIproducts suite are likely to improve the usefulness of CESIproducts. Some sources, however, consider that the use of CESIfor policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESIindicators.

a) It is recommended that once intended CESIimprovements are completed, the CESIDG Steering Committee reassess the success of the CESIinitiative.

b) It is recommended that the DG Steering Committee consider the role expected of CESIin supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act.

Management Response

The CESIinitiative, on behalf of the Government of Canada, demonstrates Canada’s ability to tell the country’s sustainability story and report to Canadians on the state of the Canadian environment. For the past four years, CESIhas demonstrated a record of organizing and disseminating information related to the environment in a scientifically credible and defensible way and tracking and reporting trends on air quality, water and greenhouse gas emissions.

Funding for the CESIinitiative was initially approved in 2003–2004 for a period of five years. Since the funding sunsets in 2008–2009, Environment Canada, in partnership with Statistics Canada and Health Canada, is seeking opportunities for the renewal of the CESIinitiative. All Management Response commitments to the Evaluation Recommendations will be dependent on the nature of the renewal of the initiative and the related available resources.

The Assistant Deputy Minister of the Strategic Policy Branch and the CESIDG Steering Committee Chair, Director General, Strategic Information Integration Directorate, accept the evaluation and its recommendations and have provided a preliminary plan to implement the recommendations within the context of a possible CESIrenewal.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESIproducts by its target audiences, the initiative needs to monitor the extent to which CESIproducts are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESIproducts. It is recommended that the CESIDG Steering Committee oversee development and implementation of a performance measurement system for the CESIinitiative. This system should include a database of target users of CESIproducts, a process for annually measuring user reach, satisfaction and use of CESIproducts, and a reporting strategy.

Response: The CESIDG Steering Committee agrees with Recommendation 1 to develop and implement a performance measurement system for the CESIinitiative. In response, within three months of the renewal of CESI, the CESIDG Steering Committee will develop and present to the Internal Services (IS) Board, for its approval, a revised CESIIndicators Logic Model and a Performance Measurement Plan that will include clear accountabilities and timelines for implementation of the performance measurement plan and associated data collections mechanisms.

Performance measurement will include:

  • An inventory of target users of CESIproducts
  • A process for annually measuring user reach, satisfaction and use of CESIproducts, (monitoring client feedback on CESIproducts through focus groups/public opinion research and through web monitoring of client statistics which will provide valuable information to ensure that the CESIproducts meet federal government policy development and decision-making needs) and
  • A reporting strategy

Recommendation 2: Several sources indicated that the target CESIusers have limited awareness of the CESIproducts and website. Given the importance of increasing awareness of the CESIproducts, it is recommended that the CESIDG Steering Committee revisit the CESIcommunications strategy to oversee, identify and implement means to increase target CESIusers’ awareness of CESIproducts.

Response: The CESIDG Steering Committee agrees with the recommendation to develop a more strategic and targeted marketing strategy. Within three months of the renewal of CESI, the CESIDG Steering Committee will present to the ISBoard a CESIEngagement Strategy that will involve collaboration with the Department’s Communications Branch to develop and implement a strategy to identify the intended target audiences and promote awareness and use of the CESIreport by Fall 2009.

This strategy would demonstrate the Government of Canada’s commitment to accountability and transparency to inform Canadians in an open and transparent manner about key environmental issues. The new Engagement Strategy will include: key principles and considerations, results of audience research, outreach objectives and proposed activities, monitoring and evaluation, budget and plan for implementation and campaign management. The strategy will address, in particular, engagement of interested partners to champion the dissemination and promotion of the CESIproducts.

Recommendation 3: In light of the significant resource investments made in the CESIinitiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESIproducts. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESIproducts suite are likely to improve the usefulness of CESIproducts. Some sources, however, consider that the use of CESIfor policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESIindicators.

a) It is recommended that once intended CESIimprovements are completed, the CESIDG Steering Committee reassess the success of the CESIinitiative.

Response: The CESIDG Steering Committee agrees with the recommendation to reassess the success of the CESIinitiative. The CESIDG Steering Committee will initiate the intended improvements in the context of the renewal of CESIand will implement a cycle of evaluation of the initiative, aligned with the renewal time frame (i.e., mid term and final evaluation in a 5-year cycle). The CESIIndicators Logic Model and Performance Measurement Plan, developed in response to Recommendation 1, will be the basis for monitoring and tracking developments towards progress.

b) It is recommended that the DG Steering Committee consider the role expected of CESIin supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness, such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act (FSDA).

Response: The CESIDG Steering Committee agrees with the recommendation to consider the role expected of CESIin supporting policy development and federal government decision making. Beyond the specific reporting mandate of the initiative, the Strategic Information Integration Directorate will apply its experience with environmental indicator development to the creation of a departmental centre of expertise to provide guidance, policy advice and statistical analysis in support of the development of new indicators, as appropriate, to strengthen the information base for 1) department-wide activities such as the Departmental Performance Report or the Program Activity Architecture, as well as 2) government-wide activities such as the annual Canada’s Performance report and implementation of the 2008 Federal Sustainable Development Act (FSDA).

The revised CESIIndicators Logic Model, Performance Measurement Plan and data collection mechanisms, developed in response to Recommendation 1, will refine and clarify the intended target audiences. One audience is: federal government managers and staff with responsibility for the Federal Sustainable Development Strategy (FSDS). As a result, the CESIindicators will be the adopted indicators in the implementation of the 2008 Federal Sustainable Development Act by December 2009 in order to meet timelines laid out in the FSDAlegislation.

1.0 INTRODUCTION

This report presents the results of the evaluation of the Canadian Environmental Sustainability Indicators (CESI) initiative conducted by Environment Canada’s Audit and Evaluation Branch between March and December 2008.

This evaluation focused on assessing the CESIinitiative’s relevance, success, cost effectiveness and design and delivery since its inception in 2005. It relied on a review of documents, including recent external studies of the initiative’s relevance and effectiveness, and key informant interviews.

This document presents the findings and recommendations of the evaluation and is organized in the following way. Section 2 provides background information on the origins of the CESIinitiative as well as a description of the structure of the current initiative. Section 3 describes the purpose of this evaluation and the methodology used, including the evaluation issues covered. Section 4 identifies the evaluation’s findings. Section 5 presents the conclusions based on the evaluation findings. Sections 6 and 7 contain, respectively, the recommendations and the management response to the recommendations.

2.0 BACKGROUND

This section presents an overview of the CESIinitiative’s origins and provides a description of its main features.

2.1 Origins of the CESIInitiative[1]

The CESIinitiative stems from two earlier initiatives. In 2000, the National Round Table on the Environment and the Economy (NRTEE) was working on ecological fiscal reform and requested funds to further pursue research on this topic. In response, the federal government allocated $9 million in the 2000 budget for two parallel initiatives: 1) the establishment of a Task Force on the Canadian Information System for the Environment; and 2) the Environmental and Sustainable Development Indicators (ESDI) Initiative.

Task Force on the Canadian Information System for the Environment (CISE)

The Task Force on the Canadian Information System for the Environment was created in October 2000 by the federal Environment Minister. The Task Force included members from a range of sectors, including consumers, university researchers, environmental activists, non-governmental organizations, municipalities, the private sector, government, and the international environmental community. Its purpose was to inform the design and implementation of an environmental information system that would provide a credible foundation for holding governments accountable; strengthen the basis for sound public policies; and enable Canadian citizens and organizations to adapt to environmental change and to play their individual and collective roles in environmental management.

At the end of its consultations, the Task Force recommended the creation of a Canadian Information System for the Environment (CISE) that would provide “environmental data to support a national set of sustainable development indicators, particularly those currently being developed by the National Round Table on the Environment and the Economy”.[2]

Environmental and Sustainable Development Indicators (ESDI) Initiative

The Environmental and Sustainable Development Indicators (ESDI) Initiative was launched by the NRTEEto develop a small set of high-level indicators that could link the environment and the economy and could be used to supplement economic indicators such as the gross domestic product (GDP). The initiative was steered by a committee formed of academics, experts on indicators and environmental accounting, business representatives, representatives of non-governmental organizations (NGOs) and staff from both Statistics Canada and Environment Canada.

In its final report in 2003, the NRTEErecommended that a small set of national natural and human capital indicators be reported annually, that Canada develop a revised System of National Accounts that would include information on natural and human capital, and that a better national network be created to monitor Canada’s environmental assets.

Following the Task Force and NRTEErecommendations, the CESIinitiative was designed to produce indicators specifically designed for political decision makers and the general public and to be reported in the annual Federal Budget statement alongside the traditional economic indicators.

2.2 Description of the Initiative

CESIGoals

The Canadian Environmental Sustainability Indicators (CESI) initiative is a program initiated by the Government of Canada to develop and report on a small set of priority environmental indicators, notably air quality, water quality and greenhouse gas emissions. It is designed to supplement traditional social and economic measures, such as gross domestic product (GDP). The indicators are intended to help provide Canadians with a better understanding of the relationships that exist among the economy, the environment and human health and well-being with respect to air quality, water quality and greenhouse gas emissions. The indicators are also intended to assist those in government who are responsible for developing policy and measuring performance.[3]

The CESIinitiative is based on the premise that Canadians need clearly defined environmental indicators -- measuring sticks that describe the health of their environment and track the results that have been achieved through the efforts of governments, industries and individuals to protect and improve the environment.[4] The CESIlogic model and planning documents identified the target audience as members of the general public, Government of Canada managers and employees, decision makers and provinces, territories and municipalities.

The initiative’s intended ultimate outcomes are:

  • Canadians recognize and use the Indicators as one source of trusted information on air, water, and greenhouse gases;
  • Knowledge is shared effectively with target audiences and stakeholders are well served nationally;
  • Canadians better understand how their actions affect the quality of the environment and human health; and
  • Political and senior decision makers are more informed and make better decisions.

The complete CESIlogic model is presented in Annex 1.

Statutory Context

The Minister of the Environment has statutory obligations under the Department of the Environment Act, the Canada Water Act, and the Canadian Environmental Protection Act, 1999 (CEPA 1999) to provide environmental information to Canadians.

In particular, the Department of the Environment Act(1971) states that:

“5. The Minister, in exercising his powers and carrying out his duties and functions under section 4, shall

  • (a) initiate, recommend and undertake programs, and coordinate programs of the Government of Canada that are designed
    • (iii) to provide to Canadians environmental information in the public interest.”

Furthermore, CEPA 1999states that:

“44. (1) The Minister shall

  • (a) establish, operate and maintain a system for monitoring environmental quality;
  • (b) conduct research and studies relating to pollution prevention, the nature, transportation, dispersion, effects, control and abatement of pollution and the effects of pollution on environmental quality, and provide advisory and technical services and information related to that research and those studies;
  • (c) conduct research and studies relating to
    • (i) environmental contamination arising from disturbances of ecosystems by human activity,
    • (ii) changes in the normal geochemical cycling of toxic substances that are naturally present in the environment, and
    • (iii) detection and damage to ecosystems;
  • (d) collect, process, correlate, interpret, create an inventory of and publish on a periodic basis data on environmental quality in Canada from monitoring systems, research, studies and any other sources;
  • (e) formulate plans for pollution prevention and the control and abatement of pollution, including plans respecting the prevention of, preparedness for and response to an environmental emergency and for restoring any part of the environment damaged by or during an emergency, and establish, operate and publicize demonstration projects and make them available for demonstration; and
  • (f) publish, arrange for the publication of or distribute through an information clearing-house
    • (i) information respecting pollution prevention,
    • (ii) pertinent information in respect of all aspects of environmental quality, and
    • (iii) a periodic report on the state of the Canadian environment.”

Funding

The funding for the initiative was set out in the 2004 federal budget and the initiative was formally approved in 2005. Treasury Board allotted $45 million for fiscal years 2005–2006 to 2008–2009 among Environment Canada ($32.5 million), Statistics Canada ($10.5 million) and Health Canada ($2 million), with an average of 53 full-time staff equivalents dedicated to the project each year. The current mandate for the initiative runs until the end of fiscal year 2008–2009.

CESIProducts[5]

The CESIinitiative reports on a yearly basis on three environmental indicators of key concern to Canadians: air quality, water quality and greenhouse gas emissions.

The air quality indicators track measures of exposure of Canadians to ground-level ozone and fine particulate matter (PM2.5). These are key components of smog and two of the most pervasive and widely spread air pollutants. Exposure to these pollutants can be harmful. Both the ozone and PM2.5 exposure indicators are population-weighted average concentrations observed at monitoring stations across Canada during the warm season (April to September).

The greenhouse gas emissions indicator tracks the annual Canadian releases of the six greenhouse gases (carbon dioxide, methane, nitrous oxide, sulphur hexafluoride, perfluorocarbons and hydrofluorocarbons) that are the major contributors to climate change. The indicator comes directly from the National Inventory Report prepared annually by Environment Canada for the United Nations Framework Convention on Climate Change.

The freshwater quality indicator uses the Water Quality Index endorsed by the Canadian Council of Ministers of the Environment to summarize the status of surface freshwater quality. Quality is assessed by examining the extent to which water quality guidelines for the protection of aquatic life (plants, invertebrates and fish) are being met at selected lake and river monitoring sites throughout Canada.

The CESIproduct suite currently[6] includes:

  • Feature ReportCESI’s main product, the Feature report, presents the latest national status for each indicator, trends over time (for some indicators), an interpretation of the indicator results, a short description of influences that may have affected them, and plans for future improvements to the indicators. Where trend information is available, the main focus is on long-term trends, not annual fluctuations. The report concludes with a discussion of how the indicators are linked, primarily focusing on the socio-economic factors influencing the status and trends associated with the indicators.
  • Highlights Report - The Highlights report shows the national situation and places more emphasis on the international context and on the important relationships that exist between the economy, the environment, and human health and well-being.
  • Data Sources and Methods Reports - These three indicator-specific reports outline and explain the underlying methods and data for each indicator as published in the Feature report. The Data Sources and Methodsreports provide technical detail and other background to facilitate interpretation of each indicator or allow others to conduct further analysis using the CESIdata and methods as a starting point.
  • Website - All CESIreports are available in web format for users to download and print on their personal computers from the CESIwebsite. Furthermore, the website offers the following additional features:
    • An explanation of what each indicator covers;
    • Maps and charts for each indicator; and
    • Additional socio-economic information in the form of a supplementary set of tables compiled to accompany the environmental sustainability indicators. The main purpose of this publication is to provide contextual information on the human activities that have influenced the environmental indicator trends.

CESIGovernance and Production Cycle

The CESIinitiative is a collaborative effort of Environment Canada, Statistics Canada and Health Canada, with input from Canadian provinces and territories. Production of the CESIproduct suite is led by Environment Canada and entails a wide range of activities including scientific research; nationwide monitoring of environmental changes; assembling the data; refining, analyzing and calculating the indicators; writing, reviewing and revising the reports and website; and planning the next steps of the CESIinitiative.

Under a tri-department Steering Committee composed of senior managers, the CESIWorking Committee, chaired by Environment Canada and Statistics Canada, oversees the work of eight working groups (Reporting and Integration, Research and Development, Information and Architecture, Air Quality, Greenhouse Gas Emissions, Water Quality, Socio-Economic Modules, and Surveys). The Chairs of these working groups all sit on the CESIWorking Committee. The following table outlines the CESIgovernance structure.

Table 1 CESIGovernance Structure

Table CESI Governance Structure

The following table outlines the annual CESIproduction cycle:

Table 2 CESIProduction Cycle

January–
April
Work planning: Highlights report and other products (content, design and structure) – outlines, mock-ups
February–
June
Website redesign (design outline, grey model)
January–
May
Indicator development improvements for upcoming year
February–
June
Data acquisition, calculation and quality assurance / quality control for each indicator
March–
June
Surveys and socio-economic data integration into indicator sections
April–
November
Writing and first draft of product suite
June–
November
Site level data (metadata and drill down capability)
November–
December
Scientific & technical and provincial review and revisions
December–
January
Management reviews, approvals and final revision
December–
February
Editing, translation and layout of product suite
November–
February
Website (design, tools, testing, content)
November–
February
Communication, engagement, marketing, briefing

3.0 METHODOLOGY

3.1 Approach

This evaluation was conducted in two separate phases. First, a review of secondary data was done. Based on the results of the secondary data analysis, key informant interviews with CESImanagers[7] were then conducted.

To conduct the secondary data analysis, a list of evaluation issues was developed, based on standard evaluation questions. For each evaluation issue, relevant documents were reviewed and key findings summarized in separate findings tables. A conclusion was then drawn on whether available secondary evidence adequately addressed each evaluation issue. A description of key secondary sources can be found in Annex 2and the full list of secondary sources reviewed is presented in Annex 3.

The large number and diversity of secondary sources available on the CESIinitiative’s relevance and performance enabled the evaluation team to adequately address most evaluation issues examined as part of this evaluation. However, two gaps were found that required additional, primary data to be collected as part of a second phase for this evaluation. The first gap pertained to the success of the initiative in achieving or being on track to achieving its expected immediate and intermediate outcomes (evaluation issue number 3). The second gap pertained to the initiative’s governance and accountability framework (evaluation issue number 7).

In order to fill these gaps, key informant interviews of selected CESImanagers were conducted. These interviews were conducted by Phoenix Strategic Perspectives Inc. on behalf of Environment Canada during the period of October 2–29, 2008. Eleven interviews in total were conducted with CESImanagers from Environment Canada, Health Canada and Statistics Canada. Interview respondents all belonged to one of the three following groups:

  • CESISteering Committee (three respondents)
  • CESIWorking Committee (four respondents)
  • CESIworking groups (four respondents)

The two main issues that were explored as part of these semi-structured interviews were questions on the achievement of CESIoutcomes, specifically the recent changes made to CESIproducts and solutions implemented to improve its efficiency, and the adequacy and effectiveness of the CESIgovernance and accountability structure. As part of these interviews, the consultant asked interview participants for any additional documentation relevant to the evaluation. One document was received as part of this process. Annex 4 includes the interview guide that was used to conduct the key informant interviews.

3.2 Evaluation Issues

The following evaluation issues were addressed as part of this evaluation:

Relevance

1. Is there a legitimate and necessary role for government in this program area or activity?

2. Is the initiative connected with societal/environmental needs?

Success

3. To what extent has the CESIinitiative achieved (or is on track to achieving) its expected immediate and intermediate outcomes for components for which Environment Canada is responsible or shares responsibility?

4. Have there been any unintended (positive or negative) outcomes? Were any actions taken as a result of these?

Cost Effectiveness

5. Are others involved in the same areas of activities and/or share similar objectives? How is duplication avoided and complementarity achieved?

6. Is the initiative delivered in the most efficient manner?

Design and Delivery

7. Is the CESIinitiative governance effective?

8. Is performance data collected against activities/outcomes? If so, is collected information used to inform senior management / decision makers?

9. Does the initiative identify clear deliverables and expected results? Is the initiative delivered as designed?

3.3 Limitations of the Evaluation

This evaluation relies on multiple sources but is limited to two lines of evidence, a review of documents and key informant interviews with CESImanagers, thereby limiting evaluators’ capacity to triangulate evaluation findings.

This limitation was, however, mitigated by the rich available documentation on the initiative’s performance. The following three documents were particularly useful:

  • International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008: This external study targeting senior-level policy makers and experts in accountability reporting addresses their perception of CESI’s relevance and usefulness. A total of 13 interviews were conducted.
  • Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007: This external study targeting industry, NGOs and members of the Canadian public addresses their perception of CESI’s relevance and usefulness. The research included four focus groups made up of members of the general public with an interest in environmental issues, and 24 one-on-one interviews with stakeholders from other target audiences, including environmental non-governmental organizations (ENGOs) and industry. Members of the Canadian public invited to participate in this study were recruited randomly among members of the adult populations of Montreal and Toronto. Participants were screened to only include those who reported having an interest in environmental issues and having looked for environmental information during the previous year.
  • Environment Canada. Strategic Information Integration Directorate (SIID). CESIReview: Issues and Challenges: In this internal review of CESIimplementation, CESImanagers identified progress towards intended CESIdeliverables, flagged challenges, and proposed potential solutions.

4.0 FINDINGS

Below are the findings of this evaluation presented by evaluation issue (relevance, success, cost effectiveness, design and delivery) and by the related evaluation questions as presented in Annex 2. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings indicate that

  • The intended outcomes or goals have been achieved or met – labelled as Achieved;
  • Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed - labelled as Progress Made, Attention Needed; or
  • Little progress has been made to meet the intended outcome and attention is needed on a priority basis – labelled as Little Progress, Priority for Attention.
  • The N/A symbol identifies items where a rating is not applicable.

A summary of ratings for the evaluation issues and questions is presented in Annex 5.

Evaluation Issue 1: Relevance
Overall Findings:
Available data show that the CESIinitiative was and remains relevant to both federal government priorities and environmental needs. Documents indicated that there is a clear role for federal government in producing environmental sustainability indicators, that an appropriate partnership was developed across federal departments and provincial/territorial governments to deliver this initiative, and that it addresses a recognized need for macro-economic environmental sustainability indicators among the Canadian public and government policy and decision makers.
Evaluation Issue: RelevanceIndicator(s)MethodsRating
1. Is there a legitimate and necessary role for government in this program area or activity?

• Demonstration of a clear/legislated mandate for federal government intervention

• Congruence with federal government priorities

• Stakeholder support for federal government intervention

• Document ReviewAchieved

Numerous sources underline the importance of the federal government’s involvement in the production of broad environmental indicators.

  • According to documents reviewed, there is a clear/legislated mandate for federal government intervention in the production of environmental indicators. The main authority for such intervention comes from the Canadian Environmental Protection Act, 1999under which the federal Minister of Environment has legal responsibility to produce and disseminate information on the state of the environment. CEPA requires the Minister to “establish, operate and maintain a system for monitoring environmental quality.”[8] In keeping with this legislated mandate, the CISETask Force recommended in 2001 that the federal government jointly sponsor with the provincial and territorial governments an institute responsible for measuring sustainability indicators.[9]
  • Budget announcements in 2000 and 2004 provide evidence of congruence of the initiative with federal government priorities at the time the CESIinitiative was created. Funding was announced in Budget 2000[10] to support collaboration between Environment Canada, the National Round Table for the Environment and the Economy (NRTEE) and Statistics Canada to develop indicators to measure environmental performance in conjunction with economic performance, while Budget 2004[11] announced funding and support for environmental technologies and remediation of contaminated sites.
  • A user needs assessment conducted by Phoenix Strategic Perspectives Inc. in 2007 also revealed support among members of the Canadian public for a strong federal role in this field. They strongly endorsed the importance of the CESIinitiative to provide Canadians with information that supports credible, informed policy decisions and good governance.[12]
Evaluation Issue: RelevanceIndicator(s)MethodsRating
2. Is the initiative connected with societal/ environmental needs?• Demonstration that the CESIinitiative addresses identified environmental/societal needs• Document ReviewAchieved

Documentation shows that stakeholders (NRTEE, Task Force on the Canadian Information System for the Environment and general users) agree with the range of indicators covered by the CESIinitiative, the need for environmental sustainability indicators to inform decision making, policy development and adaptation to environmental changes, and for a strategic approach to producing such indicators. In relying on a strategic partnership between Environment Canada, Health Canada and Statistics Canada to produce broad indicators linking environmental sustainability to the economy, the CESIinitiative addresses the expressed needs.

  • In 2001, the Task Force on the Canadian Information System for the Environment reported that Canada lacked a strategic and integrated approach to collecting environmental information, and that, as a result, Canadians lacked key environmental information. To fill this need, the Task Force recommended that the Government of Canada provide environmental information through the creation of a Canadian Information System for the Environment (CISE) in support of a national set of sustainable development indicators[13].
  • In 2003, NRTEEidentified the need for national environment and sustainable development indicators to supplement the macroeconomic indicators that currently help summarize the state of the national economy to “provide Canadians with a more robust picture of the state of our national capital”. Their list of recommended indicators included, among others, air quality, freshwater quality, and greenhouse gas emissions.[14]
  • The CISEwebsite identified environmental indicators as one of the tools essential to modern environmental management that should form the basis for effective public policies (to help identify and prioritize policy issues and help to assess the effectiveness and efficiency of environmental policies and programs)[15].
  • In 2007, CESIstaff distributed a questionnaire to 715 potential respondents aiming to measure the perceived usefulness of the 2006 and 2007 CESIpublications. A majority (74%) of the 71 individuals who responded to the survey said they intended to make use of the CESIpublications.[16]
  • Interviews with members of the general public and non-government stakeholders (i.e., industry and NGOs) as part of the assessment of user needs by Phoenix Strategic Perspectives Inc. revealed unanimous concern over environmental sustainability, and most respondents found the indicators appropriate and relevant, especially data on greenhouse gases. Respondents also believed the CESIindicators to be necessary in order to adopt best practices, establish benchmarks, foster healthy competition, enhance government credibility, enhance government accountability, and increase interconnectedness with other countries.[17]
  • Interviewees (i.e., senior level policy makers and experts in accountability reporting) from the 2008 International Institute for Sustainable Development (IISD) review of the CESIinitiative agreed with the indicators covered by the initiative but felt that data needed to be made available at a smaller, more local scale to better feed policy making and to be useful for accountability purposes. They also highlighted the need for better linkages to issues of interest to the general public and Members of Parliament (such as human health issues).[18]
  • The 2006 Standing Committee on the Environment and Sustainable Development responsible for the review of CEPA’s provisions and operation confirmed the need for “timely, accurate and accessible environmental information, integrated with socio-economic factors, to improve decision making and support progress towards sustainability” and recommended that “the government publish biennially, in electronic and hard copy formats, a comprehensive state of the environment report.”[19]
Evaluation Issue 2: Success
Overall Findings:
While the CESIinitiative has produced its intended outputs and achieved or is on track to achieve most of its outcomes pertaining to increased capacity to produce environmental indicators, no evidence was found that CESIproducts are actually being used, except by Treasury Board in its annual Canada’s Performance report and by Human Resources and Social Development Canada in its set of human well-being indicators. Attention is needed with respect to achieving intermediate outcomes targeting the use of CESIinformation by decision makers and the general public, as well as the integration of CESIresults in policy decisions. To address this limitation, CESImanagers are currently making enhancements to the CESIwebsite and products such as increased drill-down capacity and the integration of socio-economic information. The program’s information dissemination strategy also appears to require improvements, in light of target users’ generally low awareness of CESIproducts. Furthermore, the use of CESIfor government decision/policy making is seen as being somewhat dependent on a federal government commitment to specific and measurable objectives related to the CESIindicators.
Evaluation Issue: SuccessIndicator(s)MethodsRating
3. To what extent has the CESIinitiative achieved (or is on track to achieving) its expected immediate and intermediate outcomes for components for which Environment Canada is responsible or shares responsibility?

Immediate Outcomes

Evidence of:

1. more targeted partnerships and distribution of funds

2. provinces being equipped to publish their own environmental reports

3. a coordinated national approach

4. improved collaboration mechanisms

5. engagement (data exchange) by all provinces and federal departments

6. increased recognition for on-going environmental monitoring programs

Intermediate Outcomes re Practices

Evidence of:

1. CESIresults being reported to Canadians

2. government actions in response to CESIresults

3. indicator tools/approaches being used in policy/decision making (by governments and general public)

Intermediate Outcomes re Capacity

1. Evidence of increased capacity of the CESIinitiative to meet its intended outcomes (i.e., information system in place; improved and expanded monitoring systems; drill-down capacity at site, local, regional and provincial levels; improved timeliness of information dissemination)

• Document Review

• Key Informant Interviews

Progress made; Attention needed

All sources suggest that the CESIinitiative has produced its intended outputs and achieved or is on track to achieving most of its outcomes pertaining to increased capacity to produce environmental indicators The initiative, however, appears to be less successful in reaching and addressing the needs of its target clienteles (general public, non-government organizations and government policy developers and decision makers). To address this limitation, CESImanagers are currently making enhancements to the CESIwebsite and products such as increased drill-down capacity and the integration of socio-economic information. The program’s information dissemination strategy also appears to require improvements, in light of target users’ generally low awareness of CESIproducts. Furthermore, the use of CESIfor government decision/policy making is seen as being somewhat dependent on a federal government commitment to specific and measurable objectives related to the CESIindicators.

Immediate Outcomes

1) More targeted partnerships and distribution of funds:

  • Key informant interviews with CESIrepresentatives revealed that targeting of partnerships and distribution of funds are addressed satisfactorily on an annual basis. Examples provided to support positive achievements in this area include the following:
    • Funding to improve monitoring of air quality, including the improvement of existing monitors and the purchase of PM2.5monitors
    • Funding to deal with costs incurred by the provinces/territories to collect data on water quality
    • Funding to develop bilateral agreements with some provinces to improve the provision of water quality data in a timely manner
    • Work on development of an air health risk indicator with Health Canada

2) Provinces being equipped to publish their own environmental reports:

  • Nearly half of the CESIkey informant interview respondents suggested that provinces already have the capacity to publish their own environmental reports but that the extent to which this is being done varies across the provinces (Newfoundland and Québec were cited as good examples of provincial reporting). Other respondents suggested that this is not really a goal of the CESIinitiative, but that through capacity building (i.e., the initiative has helped to build monitoring capacity) and provincial-federal partnerships, provinces’ ability to publish such reports has improved.

3) Coordinated national approach and4) Improved collaboration mechanisms in placeand 5) Engagement by all provinces and federal departments:

  • CESIkey informants indicated that, at the federal level, there exists a coordinated approach and good collaborative mechanisms. The CESIinitiative was cited by several interviewees as a good example of interdepartmental coordination and collaboration, this success being attributed in part to the governance structure of the CESIinitiative.
  • All CESIinterviewees felt that federal and provincial coordination and collaboration mechanisms were good and/or improving. Specific examples provided to support this impression include the following:
    • Agreements between the federal government and the provinces have been signed for monitoring and processing of water quality information. Every province has agreed to make their water data available this year.
    • Work is being done with the provinces to ensure consistency in interpreting and reporting water quality data and there is a proposal to simplify the water quality index (WQI) to a core set of parameters that are common to most or all watersheds/monitoring stations.
    • Each year since the CESIinitiative began there has been a national CESIfederal-provincial water quality index conference/workshop in coordination with the Canadian Council of Ministers of the Environment (CCME) Water Quality Task Group.
    • The CESIair group coordinates with the National Air Pollution Surveillance (NAPS) and Canadian Air and Precipitation Monitoring Network (CAPMoN) groups through the CCMEfederal-provincial air monitoring coordinating committee.
  • There was a consensus among CESIinterview respondents that there is still work to be done in the area of water monitoring in order to ensure nationally representative water quality data.

6) Increased recognition for ongoing environmental monitoring programs:

  • Members of the general public consulted as part of the Phoenix user needs assessment showed support for environmental monitoring initiatives by stating that they actively seek information on the state of the environment and that they more readily trust environmental reports that are based on data. They stated that an initiative such as CESIis important because awareness of sustainability issues is the basis for addressing them and because such data are required for accountability and transparency purposes.[20]
  • Representatives of non-government organizations consulted as part of the Phoenix user needs assessment also reported that they routinely use environmental sustainability data. They emphasised the importance of ongoing tracking of state-of-the-environment data.[21]

Intermediate Outcomes Regarding Practices

1) CESIresults being reported to Canadians:

  • Results from the survey of recipients of the 2007 Highlights report showed that only 59% (42 of 71) of respondents were aware of the CESIinitiative prior to receiving the 2007 Highlights report and only 35% (25 of 71) of respondents recalled receiving copies of the CESI2006 Highlights report.[22]
  • Few non-government stakeholders consulted as part of the Phoenix needs assessment study were aware of the CESIproducts prior to being contacted for this research. Several members of the general public who took part in the same research stated that CESIproducts need to be promoted or advertised so that Canadians are aware of them.[23]
  • In spite of the above findings, there was consensus among CESIkey informants that the initiative is achieving and performing well in reporting CESIresults to Canadians by meeting its reporting timelines, making reports and information available, and more recently through the redesign of the CESIwebsite to make information more accessible to Canadians and to allow them to choose the level of detail they want.
  • Some CESIinterview respondents, however, felt that more could be done to increase awareness of CESIamong members of the general public. Specifically, they felt that release of CESIreports could be publicized more by putting out more media advisories.
  • Program documents indicate that communication strategies were developed for the 2005, 2006, 2007 and 2008 releases of the CESIproducts. These strategies generally recommended a “low-profile” communications approach to target stakeholders. Target stakeholders included environmental groups, parliamentarians, federal and provincial/territorial policy managers, specialists and analysts. The principal means of dissemination consisted of briefing packages distributed to policy/decision makers and other target stakeholders, posting of the CESIproducts on various websites, and posting of an official bulletin on Statistics Canada’s publication The Daily.
  • A number of improvements are currently being made to the CESIwebsite and products (to be made available with the publication of the 2008 CESIproducts) to improve their accessibility for Canadians and all potential users. These changes include the following:
    • Improved drill-down capacity (i.e., the ability to access data at site/local/regional levels).
    • Shift from lengthy, descriptive reports to a dynamic online information source (the Feature report will be discontinued in favour of a searchable online database).
    • Inclusion of trend information over time (this feature is still embryonic but will expand as sets of data are gathered for each indicator on an annual basis).
    • Better integration of socio-economic context information in the CESIproducts instead of presenting it as a separate section.
    • Redesign of the 2008 Highlights report to more explicitly target the general public by ensuring it is written in clear, understandable, and meaningful language.[24]

2) Government actions in response to CESIresults and 3) Indicators tools/approaches are being used in policy/decision making (by governments and general public):

  • Senior level policy makers and experts in accountability reporting interviewed as part of the ISSD review indicated that CESIindicators are not being used for policy development or decision making namely because they require detailed information at a scale and a level of precision that a national, contextual set of indicators such as those provided by the CESIinitiative do not provide. To address some of the perceived weaknesses which impede the indicators’ use, the IISDreport recommended:
    • reporting CESIindicators at a more disaggregated level;
    • providing more contextualization of the CESIindicators within the larger and environmental and health context of Canada;
    • providing trend information (i.e., data over a number of years); and
    • using CESIdata for accountability reporting (i.e., using CESIindicators as measures of the federal government’s achievement of specific intended outcomes and specifying which jurisdictions are accountable for which achievements).[25]
  • CESImanagers recently interviewed on this topic generally agreed with the above observations and reported that they are in the process of addressing these issues. As previously mentioned, the 2008 CESIdata will be available at the site level (drill-down capacity) using an improved website navigation capability. Data will also be better contextualized within the larger and environmental and health context of Canada by integrating socio-economic and related information into the reports themselves rather than as part of a separate document.
  • CESIkey informants, however, commented that trend information will not be available until a few years of data have been gathered systematically and that CESI’s ability to inform concrete and specific policy-related action and decision making will depend on the effective linkage of CESIindicators with socio-economic indicators in future reports.
  • Some CESIinterview respondents also explained that while the original idea behind CESIwas to develop environmental indicators similar to economic ones like Gross Domestic Product, the initiative is actually designed to provide contextual and background information to inform people and increase environmental literacy. Its current purpose is not seen to be informing program or policy development.
  • Finally, a few respondents commented that CESIcould be more effective in relation to these outcomes if the federal government committed itself to specific and measurable objectives related to the CESIindicators. In short, the point being made was that government needs to take the lead in this regard through an express willingness to use CESIinformation for decision making.
  • CESIindicators are being used by Treasury Board in its annual Canada’s Performance report and by Human Resources and Social Development Canada in its set of human well-being indicators.

Intermediate Outcomes Regarding Capacity

1) Increased capacity of the CESIinitiative to meet its intended outcomes:

Information systems in place to support the three indicators

  • The ISSD Review reported that, according to senior level policy makers and experts in accountability reporting, effective information systems were in place to support the three CESIindicators.[26]
  • CESIrepresentatives also agreed that there are effective and efficient systems in place, particularly the data collection systems, to support the three indicators. The systems in place for greenhouse gas emissions and air quality were described as good or very good, while the system for water monitoring was described as lagging a little behind the first two but improving.
  • A few CESIinterview respondents referred to the CESIwebsite and enhanced web reporting as a good information system in place to support the three indicators, and specifically to the ability to drill down to local or regional information. That said, it was also suggested that there should be more interface between the CESIwebsite and Statistics Canada’s website, especially because of the desire to integrate socio-economic information with the CESIindicators.
  • Reference was also made by CESIkey informants to Statistics Canada’s water usage surveys as part of the information system in place to support the water quality indicator. This included the agriculture water use survey, the industry water use survey, and the drinking water survey.

Monitoring systems improved and expanded in areas of air, water, greenhouse gases

  • All CESIinterview participants agreed that this outcome is being achieved, while also recognizing that there is room for improvement, especially in the area of water monitoring in order to ensure nationally representative water quality data.

Drill-down capacity (per site/local/regional/provincial)

  • Most members of the general public and non-government stakeholders (i.e., industry and NGOs) consulted as part of the Phoenix user needs assessment found that CESIinformation was not sufficiently detailed to meet their needs.[27]
  • This observation was echoed by senior-level policy makers and experts in accountability reporting consulted as part of the ISSD review. They specifically mentioned drill-down capacity as a key requirement for CESIdata to be useful for program and policy development.[28]
  • Recent CESIsenior management briefings have highlighted current plans for the CESIinitiative to transition to a new State of the Environment Reporting vision which addresses most of the concerns listed above. As part of this new vision, the CESI2008 report will provide more trends, access to site level information, better integration of socio-economic information, and enhanced web-based reporting. The web design will enable access to local and regional level information, customized information and search capacity.[29]
  • CESIkey informants confirmed that changes are currently being brought to the 2008 CESIproducts and website to address the above-mentioned concerns. While interview respondents clearly saw this as a positive development in general, nearly half indicated that the capacity to drill down, while resolving one problem (i.e., lack of access to local/regional data), gives rise to others. They identified specific problems: the fact that the provinces were reluctant to provide this kind of information as some prefer that it be accessed through their own websites and that providing these data in their raw form could potentially lead to mistaken interpretations of their meaning since there may be additional factors that have to be taken into account in order to properly interpret the data.

Ability to deliver information in a more timely fashion

  • An internal review conducted by CESImanagement at the end of 2006 revealed that although timelines and expectations of the CESIproduction process were generally met, a number of challenges were encountered, such as the late receipt of the indicators data and the resulting pressures to develop CESIproducts.[30]
  • CESIkey informants reported that the situation has improved over time but recognized that there is a structural limit to improving timeliness that will never be overcome. Specifically, it was acknowledged by all that there would always be a lag between the time the data are collected and interpreted and the time the data are published or reported. The current two-year lag was seen by all as something that could be improved, but a one-year lag was the limit of what was seen to be achievable.
  • Factors identified as having already contributed to or likely to contribute to more timely delivery of information included ceasing production of the CESIFeature report, improved reporting time on the air quality indicator, and increased use of the website as a delivery mechanism.
  • Respondents to ISSD interviews indicated that the timeliness of CESIreports is adequate and the interviews did not reveal frustration with the lack of timeliness of the data on the website.
Evaluation Issue: SuccessIndicator(s)MethodsRating
4. Have there been any unintended (positive or negative) outcomes? Were any actions taken as a result of these?• Evidence of unintended outcomes• Document ReviewN/A

No significant unintended negative or positive outcomes appear to have resulted from the initiative.

Evaluation Issue 3: Cost Effectiveness

Overall Findings:
In light of the findings outlined above on the additional activities currently under way to ensure the success of the initiative, it was deemed premature to conduct a full cost-effectiveness analysis. Rather, for the purpose of the current evaluation, cost effectiveness was measured by examining the issues of duplication, complementarity and efficiency.

The CESIinitiative appears to be delivered in a generally efficient manner. Duplication is avoided through collaboration with other federal departments and provincial governments, and complementarity is achieved through the use of the CESIindicators by Treasury Board in its annual Canada’s Performance report and Human Resources and Social Development Canada in its set of human well-being indicators.

Evaluation Issue:Indicator(s)MethodsRating
5. Are others involved in the same areas of activities and/or share similar objectives? How is duplication avoided and complementarity achieved?• Evidence of duplication/complementarity

• Document Review

• Key Informant Interviews

Achieved

While other sources of data exist, the CESIinitiative appears to have a unique mandate and to involve collaboration among the key government stakeholders.

  • By its nature, CESIis a collaborative initiative between the three participating departments (Environment Canada, Health Canada and Statistics Canada).
  • Although some duplication between CESIand other sources of data was identified by non-governmental stakeholders, the same stakeholders pointed to the need for official data on some key indicators of relevance to the Canadian public and decision makers and to the federal government as being the most appropriate provider of such data.[31]
  • CESIkey informants also identified what they consider to be other sources of indicators and data related to Canada’s environmental sustainability. These include: Environmental Signals; the National Pollutants Release Inventory (NPRI); the Clean Air website; the Greenhouse Gas Inventory; and the Canadian Council of Forest Ministers’ Criteria and Indicators of Sustainable Forest Management in Canada. While not related specifically to Canada, the Organisation for Economic Co-operation and Development (OECD) and the Intergovernmental Panel on Climate Change (IPCC) were also mentioned. In the absence of a commonly accepted framework for measuring sustainability, they observed that it is currently difficult to avoid a multiplication of sources of indicators. However, they commented that complementarity is achieved through the use of the CESIindicators by Treasury Board in its annual Canada’s Performance report and Human Resources and Social Development Canada in its set of human well-being indicators.
  • Senior level policy makers and experts in accountability reporting who were consulted as part of the ISSD review reported that the CESIindicators generally complement or are used in conjunction with other sources of data. They saw the use of CESIindicators for Canada’s performance reporting as recognition of their unique role.[32]
Evaluation Issue: Cost EffectivenessIndicator(s)MethodsRating
6. Is the initiative delivered in the most efficient manner?• Evidence of need for increasing efficiency

• Document Review

• Key informant interviews

Achieved

While some areas for improvement were identified, no evidence was found of a serious need for increasing the initiative’s efficiency.

  • The internal review conducted by CESImanagement at the end of 2006 identified the following implementation challenges:
    • the late receipt of the indicators data and the resulting pressures to develop CESIproducts on time;
    • pressures for earlier release of information;
    • ongoing improvements required of the indicators and the need to devote resources and time to longer-term research and indicator development for air and water; and
    • slowness in establishing agreements with provinces and territories to deliver the Water Quality Index.[33]
  • As a result, the following required improvements were identified: 1) streamlining processes and improving coordination of activities in order to more easily meet a very tight reporting cycle; 2) re-examining distribution of roles and responsibilities and resourcing of activities across participating departments; and 3) refining agreements and communications with provinces.
  • CESIrepresentatives interviewed in fall 2008 were generally satisfied with the efficiency of the initiative’s implementation and found that overall processes had improved. The few remaining areas for improvement identified were as follows:
    • The governance structure is very time consuming: while ensuring effective functioning, the consensus decision-making approach, the number of committees and the frequency of meetings require a lot of time and effort.
    • Production deadlines remain tight, and strict respect for deadlines thus needs to be maintained.
    • All monies targeted for the CESIinitiative must be used for CESI.
    • The need to report annually reduces the capacity to innovate in the area of indicators. Publishing every two to three years might be a better option.
Evaluation Issue 4: Design and Delivery
Overall Findings:
The CESIinitiative’s governance and accountability structure is clear and effective. Some performance data collection and reporting activities also appear to be adequate for documenting program performance and supporting decision making. These data were used by management to make design and delivery improvements, as evidenced by the changes being made to the 2008 product suite. However, the initiative lacks systematic means of effectively monitoring whether its target users are accessing and using the CESIinformation. While clear deliverables and expected results are documented and the CESIinitiative is being delivered as designed, some CESIrepresentatives disagree on the degree to which it is realistic to expect CESIto inform policy development and government decision making given that a federal government commitment to specific and measurable objectives related to the CESIindicators is currently lacking.
Evaluation Issue: Design and DeliveryIndicator(s)MethodsRating
7. Is governance of the CESIinitiative effective?

• Evidence of clear definition of roles and responsibilities

• Evidence of effective coordination, management and reporting mechanisms

• Document Review

• Key Informant Interviews

Achieved

Various sources of evidence suggest that the governance and accountability structure for CESIis clear and effective; however, a few minor factors detract from the effectiveness of the initiative’s governance, such as time-consuming procedures for consensus-making and report approvals.

  • A review of files and documents showed that the CESIinitiative has a clearly developed and up-to-date governance structure, which depicts the various relationships between the working groups, secretariat and steering committees and identifies the individuals leading these various groups.[34]
  • As well, working documents demonstrate periodic reviews of progress and regular monthly meetings among the various CESIteams and committees (Reporting and Analysis team; Indicators leads; Working Group Committee; inter-departmental) that highlight roles, responsibilities and target dates for various milestones for the development of the CESIreports. These documents provide descriptions of issues and challenges facing the CESIteams, with the assignation of roles for the delivery of agreed-on action items.[35]
  • CESIkey informant interviews also confirmed that the CESIinitiative’s governance is effective. This effective governance is attributed to two factors: the maturity of the initiative and the good functioning of the Working Committee and Working Groups.
  • Several CESIinterview participants suggested that the governance structure’s effectiveness can be in part attributed to the way it engages individuals and to the requirement for consensus. Consensus, conversely, is also listed as a factor that makes the process time-consuming.
  • Other weaknesses in the governance of CESIincluded an overemphasis on hierarchy and a time-consuming sign-off procedure where the report has to be signed off all at once instead of in stages.
Evaluation Issue: Design and DeliveryIndicator(s)MethodsRating
8. Are performance data collected against activities/ outcomes? If so, is collected information used to inform senior management / decision makers?

• Adequate performance data are available for all CESIactivities

• Evidence of decision making based on available performance data

• Document Review

• Key Informant Interviews

Progress made; Attention needed

Some performance data collection activities appear to be adequate for documenting performance and supporting decision making, but the initiative lacks systematic means of effectively monitoring whether its target users are accessing and using the CESIinformation.

  • Over the past few years, the initiative has produced numerous documents highlighting the initiative’s performance and identifying needed improvements. These include a log of the various products/events where CESIresults were specifically cited or used in meeting a departmental obligation, reviews on progress toward the achievement of CESIdeliverables, meeting summaries and records of decision, a survey of target users on the usefulness of publications, satisfaction surveys with the three CESIreports, a user-needs assessment study and interviews with potential government CESIusers.
  • The records of decision and the improvements being made to the 2008 CESIproducts and website are evidence that some of this performance information was used by senior management for decision-making purposes.
  • Despite the above-mentioned performance measurement activities, the CESIinitiative has no systematic means of effectively monitoring whether its target users are accessing and using the CESIinformation. In particular, there is no evidence that its database of potential users is representative of the target population for CESIproducts and an insufficient response was received to target user survey attempts.
  • As per commitments, Statistics Canada conducted satisfaction surveys in 2006 and 2007 to determine if the CESIreports have been successful. Survey candidates were those who had indicated an interest in participating in an evaluation of the electronic publications during a pre-solicitation for future research. In all, 64 candidates were contacted in May 2006 (11 responded) and 81 in 2007 (18 responded). The survey was posted on a website and was communicated by email. It asked about satisfaction with the three CESIreports; reasons for use; most valued features; and suggestions to improve content and presentation. It used 5-point response scales. However, the survey population, number of respondents and cell sizes were too small for the findings to be useful.
  • CESIstaff distributed a separate questionnaire with the 2007 Highlights report. It aimed to measure the perceived usefulness of the 2006 and 2007 CESIpublications. It was sent to 715 potential respondents. Only 71 recipients (10%) responded to the survey.
  • CESIinterview respondents identified some performance data strategies that could be employed in the future, including a “Your feedback” section on the CESIwebsite, measuring the number of “hits” on the CESIwebsite (and the pages visited), public opinion research on user needs and satisfaction with CESIproducts, and annual consultations/workshops with provincial partners.
Evaluation Issue: Design and DeliveryIndicator(s)MethodsRating
9. Does the initiative identify clear deliverables and expected results? Is the initiative delivered as designed?

• Documented deliverables and expected results

• Congruence between intended and reported activities, deliverables and outputs

• Document Review

• Key informant interviews

Progress made; Attention needed

The initiative identifies clear deliverables and expected results and is being delivered as designed. However, there is no consensus among CESIrepresentatives on whether CESIindicators should be expected to inform policy development and decision making.

  • The CESImain planning document clearly outlines the expected CESIdeliverables and intended outputs and results.
  • While CESIprogress reports show agreement on the expected deliverables, various sources, however, showed a lack of agreement on the appropriateness of some of the outcomes outlined in the CESIlogic model. Particularly, some CESIrepresentatives disagreed with the expectation that CESIindicators be used for policy and decision making, given the absence of a federal government strategy and clear targets and accountabilities for environmental sustainability. As previously mentioned, some CESIinterview respondents believed that the CESIinitiative was not designed to inform program or policy development in the first place, but rather that it is actually designed to provide contextual and background information to inform people and increase environmental literacy. Others argued that the CESIinitiative’s relevance, usefulness and actual use by federal policy makers would increase if the indicators could be used for accountability reporting purposes. This perspective was echoed by the senior-level policy makers and experts in accountability reporting consulted as part of the ISSD review.[36]
  • A few CESIkey informants also disagreed with the notion that equipping provinces to publish their own environmental reports should be within the purview of the CESIinitiative.
  • CESIinternal reviews and testimonies from CESIrepresentatives show that the initiative is being delivered as designed. Recent changes to the CESIdeliverables remain in keeping with the original objective of the initiative, which is to report annually on three environmental indicators of key concern to Canadians: air quality, water quality and greenhouse gas emissions.

5.0 CONCLUSIONS

Overall, evidence shows that the CESIinitiative is relevant to both federal government priorities and various stakeholders’ environmental information needs. While other similar sources of environmental sustainability data exist, the initiative appears to have a unique mandate and to involve collaboration among the key government stakeholders (i.e., Environment Canada, Health Canada, Statistics Canada, and the provinces and territories).

However, while the CESIinitiative has produced its intended outputs and achieved or is on track to achieving most of its outcomes pertaining to increased capacity to produce environmental indicators, no evidence was found that CESIproducts are actually being used, except by Treasury Board in its annual Canada’s Performance report and by Human Resources and Social Development Canada in its set of human well-being indicators. Attention is needed with respect to achieving intermediate outcomes targeting the use of CESIinformation by decision makers and the general public, as well as the integration of CESIresults in policy decisions.

In 2007, CESImanagers contracted two external studies aiming, among other objectives, to measure the awareness and perceived usefulness of CESIproducts by CESItarget users, namely, senior level policy makers, experts in accountability reporting, members of the general public with an interest in environmental issues, and representatives from ENGOs and industry. Results from these studies revealed that CESIproducts were not well known and had design flaws that limited their usefulness.

In order to address these shortcomings, and based on recommendations from these two studies, CESImanagers are currently making enhancements to the CESIwebsite and 2008 suite of products. These improvements include presenting CESIinformation in more common language, giving access to data at more specific levels (drill-down capacity), providing more trend information, and better integrating socio-economic and health context information.

One recommended change could, however, not be implemented. Several federal government policy makers and a few CESIrepresentatives argued that the use of CESIdata by federal government decision makers and policy makers would increase if it could be used for policy development and accountability reporting purposes. This is, however, seen as being dependent on the development of a federal environmental sustainability strategy and the identification of specific objectives against which the federal government would be measured using CESIindicators. No such objectives currently exist.

In light of the improvements that are currently being made to the initiative, it is too early to draw definitive conclusions on the adequacy of the CESIinitiative to meet its target users’ need. Once these changes are fully implemented, the initiative will need to demonstrate its usefulness by documenting who its target users are and by monitoring the extent to which CESIproducts are known, accessed, used and considered useful by these target users.

The initiative currently lacks a performance measurement system to effectively monitor whether its target users are accessing and using the CESIinformation. The number of responses to the two user surveys conducted by Statistics Canada was too low for the results to be meaningful. Furthermore, evidence shows that there is generally low awareness of CESIproducts among the general public and non-government stakeholders, suggesting that a more effective information dissemination strategy may also be required.

Nevertheless, the initiative is generally implemented as designed and in an efficient manner. It benefits from an appropriate and effective governance structure that reflects the interdepartmental nature of its mandate and activities. CESImanagers have also demonstrated a best results-management practice in producing meeting minutes, records of decisions and a review of CESIimplementation challenges that led to effective solutions being developed.

6.0 RECOMMENDATIONS

The following recommendations are based on the findings and conclusions of the evaluation.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESIproducts by its target audiences, the initiative needs to monitor the extent to which CESIproducts are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESIproducts. It is recommended that the CESIDG Steering Committee oversee development and implementation of a performance measurement system for the CESIinitiative. This system should include a database of target users of CESIproducts, a process for annually measuring user reach, satisfaction and use of CESIproducts, and a reporting strategy.

Recommendation 2: Several sources indicated that the target CESIusers have limited awareness of the CESIproducts and website. Given the importance of increasing awareness of the CESIproducts, it is recommended that the CESIDG Steering Committee revisit the CESIcommunications strategy to identify and implement means to increase target CESIusers’ awareness of CESIproducts.

Recommendation 3: In light of the significant resource investments made in the CESIinitiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESIproducts. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESIproducts suite are likely to improve the usefulness of CESIproducts. Some sources, however, consider that the use of CESIfor policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESIindicators.

a) It is recommended that once intended CESIimprovements are completed, the CESIDG Steering Committee reassess the success of the CESIinitiative.

b) It is recommended that the DG Steering Committee consider the role expected of CESIin supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act.

7.0 MANAGEMENT RESPONSE

The Canadian Environmental Sustainability Indicators (CESI) initiative, on behalf of the Government of Canada, demonstrates Canada’s ability to tell the country’s sustainability story and report to Canadians on the state of the Canadian environment. For the past four years, CESIhas demonstrated a record of organizing and disseminating information related to the environment in a scientifically credible and defensible way and tracking and reporting trends on air quality, water and greenhouse gas emissions.

Funding for the CESIinitiative was initially approved in 2003–2004 for a period of five years. Since the funding sunsets in 2008–2009, Environment Canada, in partnership with Statistics Canada and Health Canada, is seeking opportunities for the renewal of the CESIinitiative. All Management Response commitments to the Evaluation Recommendations will be dependent on the nature of the renewal of the initiative and the related available resources.

The Assistant Deputy Minister of the Strategic Policy Branch and the CESIDG Steering Committee Chair, Director General, Strategic Information Integration Directorate, accept the evaluation and its recommendations and have provided a preliminary plan to implement the recommendations within the context of a possible CESIrenewal.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESIproducts by its target audiences, the initiative needs to monitor the extent to which CESIproducts are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESIproducts. It is recommended that the CESIDG Steering Committee oversee development and implementation of a performance measurement system for the CESIinitiative. This system should include a database of target users of CESIproducts, a process for annually measuring user reach, satisfaction and use of CESIproducts, and a reporting strategy.

Response: The CESIDG Steering Committee agrees with Recommendation 1 to develop and implement a performance measurement system for the CESIinitiative. In response, within three months of the renewal of CESI, the CESIDG Steering Committee will develop and present to the Internal Services (IS) Board, for its approval, a revised CESIIndicators Logic Model and a Performance Measurement Plan that will include clear accountabilities and timelines for implementation of the performance measurement plan and associated data collections mechanisms.

Performance measurement will include:

  • An inventory of target users of CESIproducts
  • A process for annually measuring user reach, satisfaction and use of CESIproducts, (monitoring client feedback on CESIproducts through focus groups/public opinion research and through web monitoring of client statistics which will provide valuable information to ensure that the CESIproducts meet federal government policy development and decision-making needs) and
  • A reporting strategy

Recommendation 2: Several sources indicated that the target CESIusers have limited awareness of the CESIproducts and website. Given the importance of increasing awareness of the CESIproducts, it is recommended that the CESIDG Steering Committee revisit the CESIcommunications strategy to oversee, identify and implement means to increase target CESIusers’ awareness of CESIproducts.

Response: The CESIDG Steering Committee agrees with the recommendation to develop a more strategic and targeted marketing strategy. Within three months of the renewal of CESI, the CESIDG Steering Committee will present to the ISBoard a CESIEngagement Strategy that will involve collaboration with the Department’s Communications Branch to develop and implement a strategy to identify the intended target audiences and promote awareness and use of the CESIreport by Fall 2009.

This strategy would demonstrate the Government of Canada’s commitment to accountability and transparency to inform Canadians in an open and transparent manner about key environmental issues. The new Engagement Strategy will include: key principles and considerations, results of audience research, outreach objectives and proposed activities, monitoring and evaluation, budget and plan for implementation and campaign management. The strategy will address, in particular, engagement of interested partners to champion the dissemination and promotion of the CESIproducts.

Recommendation 3: In light of the significant resource investments made in the CESIinitiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESIproducts. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESIproducts suite are likely to improve the usefulness of CESIproducts. Some sources, however, consider that the use of CESIfor policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESIindicators.

a) It is recommended that once intended CESIimprovements are completed, the CESIDG Steering Committee reassess the success of the CESIinitiative.

Response: The CESIDG Steering Committee agrees with the recommendation to reassess the success of the CESIinitiative. The CESIDG Steering Committee will initiate the intended improvements in the context of the renewal of CESIand will implement a cycle of evaluation of the initiative, aligned with the renewal time frame (i.e., mid term and final evaluation in a 5-year cycle). The CESIIndicators Logic Model and Performance Measurement Plan, developed in response to Recommendation 1, will be the basis for monitoring and tracking developments towards progress.

b) It is recommended that the DG Steering Committee consider the role expected of CESIin supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness, such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act (FSDA).

Response: The CESIDG Steering Committee agrees with the recommendation to consider the role expected of CESIin supporting policy development and federal government decision making. Beyond the specific reporting mandate of the initiative, the Strategic Information Integration Directorate will apply its experience with environmental indicator development to the creation of a departmental centre of expertise to provide guidance, policy advice and statistical analysis in support of the development of new indicators, as appropriate, to strengthen the information base for 1) department-wide activities such as the Departmental Performance Report or the Program Activity Architecture, as well as 2) government-wide activities such as the annual Canada’s Performance report and implementation of the 2008 Federal Sustainable Development Act (FSDA).

The revised CESIIndicators Logic Model, Performance Measurement Plan and data collection mechanisms, developed in response to Recommendation 1, will refine and clarify the intended target audiences. One audience is: federal government managers and staff with responsibility for the Federal Sustainable Development Strategy (FSDS). As a result, the CESIindicators will be the adopted indicators in the implementation of the 2008 Federal Sustainable Development Act by December 2009 in order to meet timelines laid out in the FSDAlegislation.

Annex 1
CESILogic Model

Logic Model for Indicators for a Competitive Economy and Sustainable Environment

Reach

Target Audience

Members of the general public / Government managers and employees / decision makers / provinces and territories / municipalities

Ultimate Outcomes
  • “Canadians recognize and use the Indicators as one source of trusted information on air, water, and greenhouse gases.”
  • Knowledge is shared effectively with target audiences and stakeholders are well served nationally
  • Canadians better understand how their actions affect the quality of the environment and human health
  • Political and senior decision makers are more informed and make better decisions
Intermediate Outcomes

Practices

  • Results are reported to Canadians
  • Government implements actions in response to results
  • Indicators tools / approaches used in policy / decision making

Capacity

  • Information system in place to support 3 indicators
  • Monitoring systems are improved and expanded in areas of air, water, greenhouse gases
  • Drill-down capacity (per site / local / regional / provincial)
  • Ability to deliver information on a more timely basis (including monitoring)

Supporting Sponsors / Partners

Government Departments / environmental non-government organizations / technical experts / industry

Immediate Outcomes

Partner/Rules Support

  • Identification of resource / information gaps leads to targeted partnerships and distribution of funds
  • Provinces equipped to publish own environmental reports
  • Coordinated national approach (federal, federal / provincial)

Participation/Reaction

  • Improved collaboration mechanisms are in place to ensure:
    • distinct contributions
    • complementary information systems
    • coherent approach to provinces
  • All provinces and federal departments are engaged: data provided/shared/evaluated
  • Increased recognition for ongoing environmental monitoring programs
Environment Canada/Health Canada/Statistics CanadaActivities and Outputs
  • Design and development of indicators
  • Improvements to instrumentation for measuring and reporting on air quality and water quality
  • Survey work to identify gaps in air quality data and to improve understanding of water quality available for major water uses
  • Create a national source water quality index methodology
  • Develop a method for linking air quality to human health
  • Integrate the existing greenhouse gas emissions indicator into the framework for all indicators
  • Publish print and web-based reports concerning the indicators

Annex 2
Key Secondary Sources Reviewed

Environment Canada. Strategic Information Integration Directorate (SIID), List of web sites, publications and events whereby CESIproducts were distributed, cited or used, 2007This document shows tracking of the various products/events where CESIresults were specifically cited or used in meeting, a Departmental obligation, and other promotional activities including events, online presence, workshops, media / newsletter / email distribution.
Environment Canada. Strategic Information Integration Directorate (SIID), CESIReview: Issues and Challenges 2007Through an internal consultation process CESIstaff and managers have tracked progress toward the achievement of deliverables. Achievements, issues and proposed solutions are presented in tabular format.
Environment Canada. Strategic Information Integration Directorate (SIID), Meeting summaries and records of decisionsNumerous meeting summaries and records of decisions demonstrate ongoing communication of roles, responsibilities and progress.
Environment Canada. Strategic Information Integration Directorate (SIID), CESIHighlights Questionnaire 2007A questionnaire was distributed with the 2007 Highlights report. It aimed to measure the perceived usefulness of the 2006 and 2007 CESIpublications. It was sent to 715 potential respondents, 17 addresses were no longer valid, 71 responded.
Statistics Canada Client Satisfaction Surveys 2006 and 2007
  • As per its mandate under the CESIinitiative, Statistics Canada conducted satisfaction surveys in 2006 and 2007 to determine if the CESIreports have been successful. Survey candidates were those who had indicated an interest in participating in an evaluation of the electronic publications during a pre-solicitation for future research. In all, 64 candidates were contacted in May 2006 (11 responded) and 81 in 2007 (18 responded). The survey was posted on a website and was communicated by email. It asked about Satisfaction with the three CESIreports; Reasons for use; Most valued features; and Suggestions to improve content and presentation. Multiple choice answer questions used 5-point scales.

  • Population, number of respondents and cell sizes were too small for the findings to be useful.
Phoenix Strategic Perspectives Inc., CESIUser Needs AssessmentUser needs assessment and product testing qualitative study conducted by an external consulting firm. The research included 4 focus groups with members of the general public with an interest in environmental issues, and 24 one-on-one interviews with stakeholders from other target audiences, including environmental non-governmental organizations and industry.
IISD, Review of the Canadian Environmental Sustainability Indicators Initiative

Study conducted by an external research group consisting of interviews with 13 high-level policy makers and experts in accountability reporting. The study addressed the following issues:

  1. To what extent has CESIachieved (or is on track to achieve) its expected immediate and intermediate outcomes for components for which Environment Canada is responsible or shares responsibility?
  2. Are the activities still appropriate for achieving CESIgoals? If not, how might they need to change?
  3. Do the CESIactivities benefit/leverage other Environment Canada / government goals or outcomes?
  4. Based on the CESIactivities, products and analyses conducted so far and planned for the final year, what are the key gaps that would need to be filled to produce a more policy-relevant indicator suite?
  5. What are the institutional changes that are needed to reach this? Are there interim steps that can be taken?

Furthermore, the scope was specified as:

  • the activities, products, and outcomes related to CESI, but not expenditures; and
  • the overall design and delivery of CESI, from the perspective of Environment Canada.

Annex 3
Complete List of Secondary Sources

Public Documents:

Canada. Canadian Environmental Sustainability Indicators 2006. Air Quality Indicators Data Sources and Methods Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Feature Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Freshwater Indicators Data Sources and Methods Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Greenhouse Gas Indicators Data Sources and Methods Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Highlights Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2007. Air Quality Indicators Data Sources and Methods Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Feature Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Freshwater Indicators Data Sources and Methods Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Greenhouse Gas Indicators Data Sources and Methods Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Highlights Report. 2007.

Canada. Department of Finance. Budget Plan 2004.

Canada. Department of Finance. Budget Speech 2000. February 28, 2000.

Canada. House of Commons Standing Committee on Environment and Sustainable Development. The Canadian Environmental Protection Act, 1999 - Five-Year Review: Closing the Gaps. April 2007. http://cmte.parl.gc.ca/cmte/CommitteePublication.aspx?SourceId=204099

Canada. Speech from the Throne to open the First Session of the Thirty-Eighth Parliament of Canada. October 5, 2004.

Canada. Treasury Board Secretariat. Canada’s Performance 2006: The Government of Canada’s Contribution. 2006.

Canada. Treasury Board Secretariat. Canada’s Performance 2007: The Government of Canada’s Contribution. 2007.

Canadian Information System for the Environment (CISE) website: http://www.cise-scie.ca/

National Round Table on the Environment and the Economy (NRTEE). The State of the Debate on the Environment and the Economy: Environment and Sustainable Development Indicators for Canada. 2003.

Pinter, Laszlo, Peter Hardi and Peter Bartelmus. Sustainable Development Indicators: Proposals for a Way Forward. Discussion Paper Prepared Under a Consulting Agreement on Behalf of the United Nations Division for Sustainable Development (New York, 13-15 December 2005). International Institute for Sustainable Development (IISD). 2005.

Sustaining the Environment and Resources for Canadians website: http://environmentandresources.gc.ca/default.asp?lang=En&nav=B3BE4E55-00

Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions. October 2001.

Internal Documents:

Canadian Environmental Sustainability Indicators (CESI). CESIGovernance Structure. July 4, 2007.

Canadian Environmental Sustainability Indicators (CESI). List of web sites, publications and events whereby CESIproducts were distributed, cited or used. September 27, 2007.

Environment Canada. CESILogic Model. 2005.

Environment Canada. CESIWater and Air Quality Deliverables Status. January 26, 2007.

Environment Canada. CESIWater Quality Indicators Mid-Term Progress Report for 2007. Draft for Discussion. Version 4. June 27, 2007.

Environment Canada. Communications Advisory note: Publication Release. The Canadian Environmental Sustainability Indicators (CESI) Report 2008–2009.

Environment Canada. Communications Advisory note: Publication Release. The Canadian Environmental Sustainability Indicators (CESI) Report 2007–2008.

Environment Canada. Communications Advisory note: Publication Release. The Canadian Environmental Sustainability Indicators (CESI) Report 2006–2007.

Environment Canada. Engagement Strategy: Canadian Environmental Sustainability Indicators Initiative 2007-2008.

Environment Canada. Strategic Information Integration Directorate (SIID). Canadian Environmental Sustainability Indicators (CESI): Where Are We now? (PowerPoint Deck) July 17, 2007.

Environment Canada. Strategic Information Integration Directorate (SIID). Environmental Sustainability Indicators – Modernizing the State of the Environment Reporting (PowerPoint Deck for Strategic Integration Board). February 14, 2008.

Environment Canada. Strategic Information Integration Directorate (SIID). CESIHighlights Targeted Distribution (December 13, 2007) – Questionnaire Findings. 2007.

Environment Canada. Strategic Information Integration Directorate (SIID). Next Generation. State of Environment Reporting(PowerPoint Deck). March 27, 2008.

Environment Canada. Strategic Information Integration Directorate (SIID). CESIReview: Issues and Challenge. January 26, 2007.

Environment Canada. Strategic Information Integration Directorate (SIID). Review of CESI2005: Areas for Discussion.

Environment Canada. Strategic Information Integration Directorate (SIID). How EC Works On CESI. Items for Discussion, EC Indicator Team Leads Meeting. January 31, 2006.

International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008.

Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007.

Statistics Canada. 2007 Evaluation: Canadian Environmental Sustainability Indicators Reports – Preliminary Tabular Results (May 30, 2007–June 7, 2007). June 8, 2007.

Statistics Canada. Benchmarking of Statistics Canada CESIPerformance Against Commitments. January 26, 2007.

Statistics Canada. Draft Report: Tabular Results for 2006 Environment Client Satisfaction Assessment Survey for CESI2005. Canadian Environmental Sustainability Indicators Reports. May 2006.

Statistics Canada. Suggestions to Improve the CESIReport and Development Process for 2007. January 18, 2007.

Annex 4
Key Informant Interview Guide

The Audit and Evaluation Branch of Environment Canada is conducting an evaluation of the Canadian Environmental Sustainability Indicators (CESI) initiative. The purpose of this evaluation is to assess the relevance, success, cost effectiveness and design and delivery of the initiative since its inception in 2005 in order to meet Treasury Board requirements. A first phase of this evaluation was completed this spring and consisted of a review of existing documentation.

As part of the second phase of this research, we are conducting key informant interviews with managers involved in CESI’s implementation in order to obtain your perspectives on the issues being examined.

The information collected as part of these interviews will be held in strict confidence and will not be attributed to specific individuals; rather, responses will be rolled up into a higher level assessment of the initiative. The interview is expected to take between 45–60 minutes.


1. Could you briefly describe your role and responsibilities with respect to the CESIinitiative?

2. I’m going to identify a number of outcomes identified for the CESIinitiative, and for each one I’d like to know to what extent you think that CESIis achieving each particular outcome, and why you say that?

  • Reporting results to Canadians
  • Government implementing actions in response to results
  • Indicators tools/approaches being used in policy/decision making (by governments and/or the general public)
  • Information systems in place to support the three indicators
  • Monitoring systems improved and expanded in areas of air, water, GREENHOUSE GAS EMISSIONS
  • Drill-down capacity (per site/local/regional/provincial)
  • Ability to deliver information on a more timely basis (including monitoring)
  • Identification of resources/information gaps leading to targeted partnerships and distribution of funds
  • Provinces being equipped to publish their own environmental reports
  • Coordinated national approach (federal, federal/provincial)
  • Improved collaboration mechanisms in place

3. Recent CESIsenior management briefings have highlighted current plans for CESIto transition to a new State of the Environment Reporting vision. Are you aware of this new vision? If so, could you please explain what it consists of?

4. As part of senior management briefings, it was suggested that improvements were being made to the 2008 CESIproducts in order to increase their usefulness and accessibility. Were you aware of this?

5. Could you please describe what these intended changes entail?

6. I’m now going to identify some perceived gaps or weaknesses in CESIidentified by users among the general public and federal decision makers. Using a 5-point scale, where 1 means not effective at all and 5 means very effective, I’d like to know to what extent you think recent changes in the last year or planned changes will be effective in addressing them.

  • Availability of data at more local scales/drill-down capacity (e.g., at the site, local, regional and provincial levels)
  • Better linkages to issues of interest to the general public and Members of Parliament (e.g., human health issues, linkage with socio-economic indicators)
  • Increased usefulness of CESIindicators in accountability reporting (i.e., to assist the federal government in reporting on the achievement of specific goals or policy objectives)
  • Need for better navigability of the website
  • Need for increased awareness of CESIproducts by the general public
  • Need for official government indicators that would serve as THE source of official data from the federal government.

I’d now like to explore issues related to CESI’s governance and accountability framework. In order to help guide the discussion, please refer to the organization chart presenting the CESIgovernance structure, appended to the interview guide.

7. In your opinion, to what extent is the governance of the CESIinitiative effective? Please explain.

8. In your opinion, are there any (other) changes that should be made to the CESIinitiative that would have a direct impact on improving its efficiency or effectiveness? If so, what?

9. To your knowledge, do other sources of indicators and data related to Canada’s environmental sustainability currently exist? If so, what are they? How is duplication avoided and complementarity achieved between CESIand this/these other indicator(s)?

10. How do you plan to measure user satisfaction and usefulness of the CESIproducts in the future?

11. Do you have any other observations to add regarding CESI’s relevance, success, or effectiveness?

12. Should the need arise, would it be possible to contact you to revisit any of the issues explored during the interview? Should we feel the need to do this, we would only take a few additional minutes of your time.

THANK YOU VERY MUCH FOR YOUR TIME AND THOUGHTFUL FEEDBACK.
IT IS VERY MUCH APPRECIATED.

Annex 5
Summary of Findings

Evaluation Question (EQ)AchievedProgress Made, Attention NeededLittle Progress, Priority for AttentionNot Applicable
Relevance:
EQ1 Role of government in CESIX   
EQ2 Connection with societal and environmental needsX   
Success:
EQ3 Achievement of outcomes X  
EQ4 Unintended outcomes   X
Cost Effectiveness:
EQ5 Avoidance of duplication and achievement of complementarityX   
EQ6 Increasing efficiencyX   
Design and Delivery:
EQ7 Effectiveness of CESIgovernanceX   
EQ8 Performance measurement collection and use in decision making X  
EQ9 Clear deliverables and expected results and program delivered as designed X  

[1] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008.

[2] Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions, October 2001, pp. 9–13.

[3] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 4.

[4] Ibid, p. 5.

[5] http://environmentandresources.ca/default.asp?lang=En&n=2102636F-1

[6] The 2008 CESIproduct suite includes modified features that will be described later in the findings section of this report. Since this new suite has not yet been published, the description provided herein refers to the 2007 CESIproduct suite.

[7] For the purpose of this study, managers include directors general, directors, chiefs and managers.

[8] Canadian Environmental Protection Act, 1999. Section 44(1)(a).

[9] Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions. October 2001. p. 14.

[10] Canada. Department of Finance. Budget Speech 2000. February 28, 2000. pp. 14–15.

[11] Canada. Department of Finance. Budget Plan 2004. pp. 142, 161.

[12] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. p. 25.

[13] Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions. October 2001. pp. 4–6.

[14] National Round Table on the Environment and the Economy (NRTEE). The State of the Debate on the Environment and the Economy: Environment and Sustainable Development Indicators for Canada. 2003. pp. xviii, 15.

[15] Canadian Information System for the Environment (CISE) website: http://www.cise-scie.ca/.

[16] Environment Canada. Strategic Information Integration Directorate (SIID). CESIHighlights Targeted Distribution (December 13, 2007) – Questionnaire Findings. 2007.

[17] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. pp. 6, 9–10.

[18]International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 24.

[19] Canada. House of Commons Standing Committee on Environment and Sustainable Development. The Canadian Environmental Protection Act, 1999 - Five-Year Review: Closing the Gaps. April 2007. http://cmte.parl.gc.ca/cmte/CommitteePublication.aspx?SourceId=204099

[20] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. p. 9.

[21] Ibid., pp. 28–33.

[22] Environment Canada. Strategic Information Integration Directorate (SIID). CESIHighlights Targeted Distribution (December 13, 2007) – Questionnaire Findings. 2007.

[23] Phoenix Strategic Perspectives Inc., pp. 25, 34.

[24] Environment Canada. Strategic Information Integration Directorate (SIID). Next Generation. State of Environment Reporting (PowerPoint Deck). March 27, 2008; CESIkey informants.

[25]International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 27.

[26] Ibid, p. 22.

[27] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. pp. 30–31.

[28]International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 22.

[29] Environment Canada. Strategic Information Integration Directorate (SIID). Next Generation. State of Environment Reporting (PowerPoint Deck). March 27, 2008.

[30] Environment Canada. Strategic Information Integration Directorate (SIID). CESIReview: Issues and Challenge. January 26, 2007.

[31] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. p. 25.

[32]International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 26.

[33] Environment Canada. Strategic Information Integration Directorate (SIID). CESIReview: Issues and Challenge. January 26, 2007.

[34] Canadian Environmental Sustainability Indicators (CESI). CESIGovernance Structure. July 4, 2007.

[35] Environment Canada. Strategic Information Integration Directorate (SIID). CESIReview: Issues and Challenge. January 26, 2007.

[36]International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 27.