Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2010-2011 > Evaluation Review of the Clean Air Agenda Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings

Evaluation Review of the Clean Air Agenda Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings

Previous page | ToC | Next page

• 4.0 Findings
  • 4.1 Relevance
  • 4.2 Performance

4.0 Findings

The findings of this evaluation review are presented by evaluation issue (relevance and performance) and by the related evaluation questions. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings from the program-level evaluation and performance data indicate that:

• the intended outcomes or goals have been achieved or met — labelled as Achieved;
• considerable progress has been made to meet the intended outcomes or goals, but attention is still needed — labelled as Progress Made, Attention Needed; or
• little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis — labelled as Little Progress, Priority for Attention.

The N/A symbol identifies items where a rating is not applicable, and the ~ symbol identifies outcomes achievement ratings that are based solely on subjective evidence.

Given the limitations noted in this report, the ratings should be considered tentative. In some cases (e.g., evaluation question 3), no rating is given as the evaluation questions could not be assessed based solely on findings from program evaluations.

Top of Page

4.1 Relevance

As part of the Clean Air Agenda, the Adaptation Theme represents a federal investment in research, tools, partnerships and standards to help reduce risks to Canadians, communities and infrastructure resulting from climate change and air pollution. The evidence demonstrating the relevance of this Theme, based on data collected during the individual program evaluations, indicates clear alignment with both federal and departmental priorities, and demonstrates program coverage of a range of identified environmental climate change and air quality needs.

The alignment with federal government priorities was examined in each of the individual program evaluations, typically through a document review and interviews to assess the extent to which the program’s individual rationale was linked to specific federal priorities. As noted in the introductory section, the Adaptation Theme was developed with the intent of addressing climate change adaptation issues in three federal priority areas: the North, human health, and infrastructure. The evidence from the evaluation reports indicates that the Adaptation Theme is aligned with these three federal priority areas. For example,

• evidence presented in the evaluations of the Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program and the Climate Change and Health Adaptation in Northern and Inuit Communities Program found that the programs were aligned with the priorities of the federal government with respect to northern development and such commitments as the International Polar Year, the United Nations Framework Convention on Climate Change, and the Intergovernmental Panel on Climate Change;
• evidence presented in the Air Quality Health Index Program (AQHI) evaluation, the draft Heat Resiliency Program evaluation and the draft Pilot Infectious Disease Impacts and Response System (PIDIRS) evaluation pointed to alignment with federal priorities with respect to human health through the development of tools, such as a nationally-consistent air quality index based on health-related information, the development of pilot projects to provide alert and response information to the public during extreme heat events, and research on the public health risks associated with infectious diseases and climate change; and
• evidence presented in the draft Improved Climate Change Scenarios (ICCS) program evaluation noted alignment with the federal priority with respect to protecting infrastructure through research and related activities on hazards and climate extremes that upgrades existing Canadian building codes and standards to include the impacts of climate change on existing infrastructure.

Top of Page

Furthermore, there was a lack of significant evidence of duplication of other programs or overlap with other jurisdictions in any of the program evaluations.

Alignment with specific departmental priorities was also explored in each individual evaluation. The individual evaluations contain details on the specific departmental priorities that are not reproduced here as they refer to program-specific details. All Adaptation programs were found to be aligned with priorities within each of their participating departments. For example:

• the Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program is aligned with TheNorth, The People and The Land INAC priorities;
• the AQHI and the ICCS are aligned with EC priorities related to weather and environmental predictions and services to reduce risks and contribute to the well-being of Canadians;
• the AQHI, Climate Change and Health Adaptation in Northern and Inuit Communities and the Heat Resiliency programs are aligned with HC’s priorities with respect to providing health-related information on air and other environmental pollutants and with HC’s Sustainable Development Strategy; and
• the PIDIRS program is aligned with the Public Health Agency of Canada’s priorities on Environment Issues.

Top of Page

A rating of N/A (not applicable) was given for question 3 as it was not feasible to assess the extent to which the Theme is connected to key environmental climate change and air quality needs based solely on program evaluation findings. An assessment of the extent to which the current Theme is connected with these key needs requires information on the full range of potential climate change and air quality needs that might be addressed through adaptation policy and programs. At this point, the evidence contained in the individual evaluations provides a description of the programs’ connections to key climate change and air quality needs, such as:

• data and information on global and regional climate models, hazards and climate change scenarios to facilitate research activities in the larger climate science community, both in Canada and internationally;
• alert and response information as a result of extreme heat events, given the deaths associated with recent heat events in the American midwest and Europe;
• climate change information and adaptation tools for communities in the North, given the significant potential impacts of climate change on northern communities;
• tools to address the impact of air quality on human health, given the growing scientific consensus that air quality can impact a range of human health issues;
• findings from NRCan’s Baseline Survey, directed at that department’s target audiences, indicating that seven in ten business respondents felt climate change would have a mainly negative impact on their organization and that the impacts identified by these respondents were those areas addressed by the current NRCan programming; and
• research designed to understand the public health implications related to infectious disease arising from climate change, given the growing body of research linking infectious diseases and climate change.

Top of Page

4.2 Performance

It is very early in the implementation of the programs within the Adaptation Theme to provide a valid assessment of thematic performance. A frequent limitation noted in individual program evaluations was that these new programs were too early in their implementation to demonstrate evidence of impact and achievement of outcomes. While the programs within the Adaptation Theme appear to be progressing toward their intended outcomes, the program evaluations provided limited evidence of the achievement of outcomes.

The evidence presented in this section refers to the extent to which the Adaptation Theme has met the outcomes identified in the thematic logic model. Each individual evaluation addressed a specific set of thematic outcomes based on alignment with the program’s goals and objectives. It was assumed at the outset of the thematic evaluation process that no individual program evaluation would address every element in the logic model, as some outcomes were not relevant or applicable to each individual program. Taken together, and provided with enough time to implement expected activities and outputs, the programs should address all outcomes articulated in the thematic logic model.

A common limitation in conducting program evaluations was the recent implementation of the programs. As such, there was limited evidence to assess the evaluation of outcomes, since programs had only begun to implement intended activities and reach external stakeholders. Furthermore, this review is based on evaluations at varying stages of completion.¹⁰

Immediate Outcomes

There was initial evidence to indicate that the expected immediate outcomes related to increasing availability and access to products, increasing capacity and greater collaboration had occurred in cases where the program had begun to implement program activities.

Top of Page

Increased availability of / access to information and products

Evidence on access to websites and website usage, in addition to other qualitative information on the increased availability of products and technical expertise, appears to indicate that the programs within the Adaptation Theme have begun to increase the availability of and access to adaptation information and products. The following points provide more detailed information:

• the draft ICCS evaluation found that data downloads and data requests from the program’s websites have been generally increasing since 2007. Furthermore, evidence demonstrates that awareness and availability of and access to climate change information and research among target audiences have increased thanks to the availability on program websites of publications by program scientists as well as training sessions on how to use climate change scenarios data and hazards and climate extremes information;
• the AQHI evaluation noted that the federal AQHI website is operational and is available to all members of the public. In addition, website hit analysis of the Toronto pilot of the AQHI revealed increasing interest in seeking out AQHI information;
• the formative Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program Evaluation found early evidence that the program had brought technical expertise into northern communities and that projects were developing climate change adaptation information that was accessible and relevant to these communities;
• the evaluation of Climate Change and Health Adaptation in Northern/Inuit Communities found early evidence of increased access to tools and information through program-developed information materials and publications, videos and other information materials designed to increase awareness of climate change risks developed by northern communities and organizations; and
• the Heat Resiliency Program evaluation noted that information on the program has been presented at various conferences across Canada, and that the program has worked with the Weather Network to prepare heat-related vignettes. In addition, the program disseminated heat-health information to the program pilots and developed an outreach strategy for dissemination after 2011, in consultation with two external program advisory committees.

Top of Page

Increased capacity to conduct and apply science

Evidence on increased capacity to conduct and apply science presented in the evaluation reports indicates that the programs within the Adaptation Theme had generated both new science and tools and techniques to apply climate change adaptation science, such as those noted in the following examples:

• evidence in the formative Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program evaluation indicates that northern communities are assessing climate change risks and opportunities and defining adaptation priorities; and
• evidence from the ICCS evaluation indicates increased use of program websites and target models, although the evaluation also noted that increased capacity for target populations to conduct climate change impact and adaptation research largely depended on the target group. Interviewees reported that availability of climate change models and training sessions provided by the program contributed to increased capacity among scientists to conduct their own climate change impacts and adaptation research. Documentation, however, demonstrated that community leaders still require the expertise of program officers to use and apply specialized climate information, as well as to understand this information in a broader context when developing strategies to reduce vulnerability to climate change.

Greater collaboration to address the issue of climate change

The evidence presented in the individual evaluations appears to indicate that the programs within the Adaptation Theme have been successful in generating initial collaborative approaches to address climate change adaptation issues. For example:

• the creation of two external advisory committees to the Heat Resiliency Program to provide strategic direction on the main components of the Heat Project and identify information gaps and research topics, in addition to the set-up of four advisory committees in each pilot community to support the collaboration of local/regional stakeholders and partners implementing the HARS pilot project;
• provincial agreements to implement the AQHI in nine of ten provinces, including partnerships with NGOs within these provinces, and a partnership agreement with the Weather Network to roll out the AQHI across the majority of metropolitan regions in Canada;
• collaboration with various Aboriginal groups (e.g., Assembly of First Nations Canada) and the INAC to support work under the Climate Change and Health Adaptation in Northern/Inuit Communities Program; and
• research collaboration with other government departments, provincial organizations, research organizations and universities, non-governmental organizations, international partners and industry was noted in the draft ICCS evaluation.

Top of Page

In addition, evidence presented in the Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program formative evaluation indicates that a wide range of stakeholders such as scientists, consultants, experts and communities are collaborating on climate change adaptation projects. These collaborations were noted to have helped create a strong network of researchers and communities interested in climate change adaptation.

Intermediate Outcomes

Evidence of intermediate outcomes was not widely reported in individual program evaluations, given the limitations related to early program implementation noted previously. The evidence that was presented was generally speculative and based on indirect lines of evidence (e.g., program staff commenting on the use of adaptation tools by other groups). This limited information should not be interpreted as limited or poor performance; rather, there has simply not been sufficient program implementation to achieve anticipated impacts related to complex issues such as behavioural change or community capacity.¹¹ Finally, as noted elsewhere in the report, the findings of the Adaptation Theme Evaluation Review are based on evaluations at varying stages of completion.¹² The following findings are therefore presented at a general level and are based on an assessment of what is currently available.

Increased use of information and products

At present, the evidence on increased use of information and products points to increased use among groups with a clear interest in using or incentive to use adaptation products and information.

• For example, the draft ICCS evaluation found that, while the program’s websites have been increasingly accessed over the last two years, it was difficult to determine how they were being used. While scientific researchers reportedly accessed and made use of the climate change data and information, other target groups with less technical or scientific capacity (e.g., engineers, municipalities, resource management and risk planning agencies) may not be able to use this information without consultation with climate experts.
• Communities identified in the formative Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program evaluation were assessing climate change risks and opportunities and defining adaptation priorities. While projects were developing adaptation tools, there were no processes in place to track how they were being used. The evaluation did not find evidence to support whether planning decisions were being based on identified risks and opportunities, or if climate change information or adaptation information was being integrated into planning and decision-making processes. These issues will be investigated further in the summative evaluation of the program.
• Respondents in the National Climate Change Adaptation Baseline Survey, such as non-federal government or business people who are part of NRCan’s program stakeholders, also indicated that a lack of organizational climate change expertise was one of the main barriers to integrating climate change adaptation issues into decision-making.
• Finally, interviewees in the AQHI evaluation indicated that increased use of the AQHI was particularly notable for at-risk groups that were using the information to guide their actions with respect to outdoor activity.

Top of Page

Taken together, the results appear to indicate that the current use of adaptation information and products is limited to groups who have an immediate interest in climate change impacts (e.g., scientists or individuals who are at risk of adverse health impacts) , and that there are still opportunities to broaden the use of information and products. Additional evaluative work will be required to fully and reliably assess this outcome.

Increased awareness of risks

Evidence from the program evaluations suggests that there has been increased awareness of risks associated with the impacts of poor air quality and climate change; as with the increased use intermediate outcome, however, this is mainly limited to specific target populations (e.g., climate scientists, individuals at-risk to air pollution).

Results of the National Climate Change Adaptation Benchmark Survey, however, did reveal an area associated with risk not explored in the individual evaluations. The survey results suggested that many non-federal government and business partners do not distinguish between the federal approach to climate change adaptation (i.e., policies to reduce risks to Canadians as a result of climate change) and the federal approach to climate change mitigation (i.e., policies to reduce GHG emissions and/or air pollution). As a result, external stakeholders may still expect federal adaptation policies will address climate change mitigation issues rather than issues associated with risk management, such as planning for extreme weather events or potential changes to building codes.

Increased capacity to act on climate change and air quality issues among target populations

Evidence from individual program evaluations suggests that there is a growing capacity to act on climate change issues and air quality issues among target populations, for example:

• communities in the North that are assessing climate change risks and opportunities and defining adaptation priorities;
• individuals at-risk to poor air quality taking action to reduce risk; and
• regional and municipal partners in both government and industry developing adaptation strategies and plans within their own organizations (e.g., reductions in infrastructure vulnerability, development of extreme heat program, changes to regional operations to address increased risk for wildfires).

Top of Page

Capacity, however, may be assessed differently by different target populations. For example, rural community capacity to adapt to decreased precipitation or northern community adaptation to rising sea levels is qualitatively and quantitatively different from the capacity of an individual to change behaviour based on extreme heat or air quality information. As with the other intermediate thematic outcomes, while the early evidence points to positive steps taken towards increased capacity, additional evaluative work will be required to fully and reliably assess this outcome.

Additional approaches to adapt to climate change are developed in targeted areas by appropriate jurisdictions

Evidence from program evaluations indicates that, at present, it is too early to assess this outcome.

Top of Page

There is generally limited information from which to draw thematic conclusions on efficiency and economy at this point. The evidence from evaluations is either not yet collected, or is based on qualitative data due to challenges linking financial information to outputs (efficiency) and outcomes (economy). There is not yet sufficient evidence from program evaluations to draw reliable conclusions.

Evidence from the individual program evaluations and thematic financial reporting indicates that programs are generally being implemented as planned, although unique challenges were highlighted in each evaluation.

The Adaptation Theme reported planned and actual expenditures through the Clean Air Agenda’s Horizontal Management, Accountability and Reporting Framework (HMARF), the results of which were reported in two consecutive Environment Canada Departmental Performance Reports (reported in Table 3). The financial data indicate that the first year of implementation was, in most cases, fiscal year 2008-2009, highlighting the late start to implementation noted in a number of the individual program evaluations. The first full year of implementation of the Theme (2008-2009) saw the majority of allocated resources used, from roughly 68% of planned resources used by the Improved Climate Change Scenarios Program¹³ to 88% of planned resources used by the Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program. These results indicate that the programs within the Adaptation Theme seem to be moving towards planned implementation; however, it is too early to provide a complete assessment, since resource use by all programs for the 2009-2010 fiscal year was not available at the time of the Adaptation Evaluation Review.¹⁴

Table 3. Current Adaptation Approved vs. Actual Spending (in thousands)

The evidence from program evaluations indicates that programs were generally being implemented as planned, including evidence that

• the AQHI is on track to being implemented as planned, with the important caveat that there are continued risks to implementation in Ontario, Quebec and, in particular, Alberta;
• the Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program and the Climate Change and Health Adaptation in Northern/Inuit Communities Program are generally on track to being implemented as planned, given the funding of projects supporting the program objectives to assess and identify climate change risks and opportunities and to develop climate change adaptation plans;
• the ICCS is generally on track to being implemented as planned, although it is not clear whether or not complete implementation will take place within the program timeframe; and
• the Heat Resiliency Program is on track to being implemented as planned, has established a framework to manage its activities and progress, and has developed a strategic plan, project implementation plan and evaluation plan.

Generally, interviewees indicated that the anticipated theme governance and accountability structures were in place and were functioning. However, several areas for improvement were noted.

Top of Page

• Interviewees noted that the Adaptation Theme was developed later in the Clean Air Agenda process (i.e., after the other themes were developed) and was not as cohesive as other CAA Themes. The term “cobbled together” was frequently used by Theme stakeholders to describe the Adaptation Theme. Internal stakeholders also noted, however, that these programs must be considered an initial set of adaptation programs as the policy direction to address adaptation has not been finalized. There was also a wide variation in the scope and reach of the individual programs; for example, some programs addressed region-specific issues (e.g., the North) and relied on extensive community participation, while others were traditional science-based programs that collected and reported global climate change data. In addition, one major program under the Adaptation Theme, the AQHI, was not strictly speaking an adaptation program, since its goal was to promote individual health based on meteorological information rather than develop strategies to reduce risk based on climate change information.
• Despite this noted lack of cohesiveness, program managers in all departments were unanimous in their comments regarding the effective functioning of the Theme governance structure, noting the following points:
  • The Theme has a Directors General Theme Management Committee that meets regularly to provide updates on program progress and to discuss CAA reporting and governance issues. The DGMC meetings are attended by all departments, though it was also noted that the meetings are usually attended by program staff rather than senior managers.
  • The Chair of the DGMC has moved from department to department on an annual basis. This practice was established to ensure equal partnership and was lauded by all interviewees.
  • The DGMC could be even more effective by focusing on forward-looking Adaptation policy and science issues, rather than the operational and administrative issues that were more commonly discussed.
  • There was initial evidence of cooperation and coordination between programs, such as the cooperative work between INAC and HC on addressing adaptation issues for northern communities.
• The main challenge noted by interviewees was the financial and non-financial reporting burden required as a Theme. However, it was also noted that despite the resources used to report as a Theme, the reporting process had facilitated the dissemination of status updates on financial and non-financial performance to senior management across departments and had helped increase the visibility of the Theme and its objectives.
• Interviewees also generally noted that they were still unclear about the future of the Adaptation Theme, since funding is set to expire in March 2011 and formal plans for continued work have not been developed.¹⁶

Top of Page

There is limited evidence to report for the Adaptation Theme regarding lessons learned or best practices, as they were generally program-specific, perhaps because of the noted variation in the scope and reach of the individual programs. Specific suggestions made during the course of the evaluations are available in each individual evaluation report. A number of lessons learned arising from some evaluations that appear to address thematic issues do bear noting, particularly those regarding the importance of engaging with external stakeholders and recognizing that adaptation is one of many competing priorities and that sustained effort on the part of all partners will be required to address the issues.

• Stakeholders in the evaluation of the Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities Program identified the following lessons learned that would help improve the federal government’s approach to adaptation programming:
  • repeat messaging about climate change is needed;
  • communities have many priorities other than climate change to address;
  • jurisdictional barriers can make it difficult to conduct research;
  • adaptation takes time; and
  • communities need additional resources to implement their adaptation plans.
• Program management and staff for the AQHI indicated that using local partners to act as outreach agents on behalf of the program was an effective approach, as it allowed the program to access sensitive and targeted populations that it would not have otherwise been able to reach and to widen the network of government and NGO partners engaged in promoting and developing the AQHI.
• Program management and staff for the ICCS noted the following best practices:
  • build in-house science capacity. Recruiting new staff allows for knowledge transfer from senior-level scientists and helps the program to achieve its goals and expected activities for the funded period;
  • partner and link with the Canadian climate research community. One of the program’s strengths are the synergies created with research networks in universities, regional research groups and regional Environment Canada staff, which enhance internal research capacity and reduce duplication of effort; and
  • implement training workshops.The workshops are designed to teach researchers and other target users how to access and utilize the data available on the program’s website.¹⁷

Top of Page

¹⁰ As noted earlier, there were only four completed evaluations at the time of the Evaluation Review, of which only two had been approved by their respective Departmental Evaluation Committees. Evaluation Review findings also were derived from the preliminary findings of two other ongoing program evaluations and a stakeholder survey used for performance measurement purposes.

¹¹ Annemoon et al. noted in the article “Evaluating the Effectiveness of Air Quality Interventions” (Journal of Toxicology and Environmental Health, Part A, 71: 583–587, 2008) “considerable challenges in assessing the health impact of air quality regulations can be anticipated. For example, regulations take effect at different times and are implemented at multiple levels of government, asking for evaluations at different time scales and at the national, regional, or local level. In addition, other changes may take place as well, for example, in economic activity or at the personal behavior level, such as people staying indoors when pollution levels are high. Finally, other causes of air pollution-related adverse health effects need to be taken into account. These challenges become more prominent when regulations are implemented over a long time period and improvements in air quality and adverse health effects are not seen immediately, increasing the chance for confounding by other risk factors.”

¹² As noted earlier, there were only four completed evaluations at the time of the Evaluation Review, of which only two had been approved by their respective Departmental Evaluation Committees. Evaluation Review findings also were derived from the preliminary findings of two other ongoing program evaluations and a stakeholder survey used for performance measurement purposes.

¹³ Funding for this program was not received until the fall.

¹⁴ The FY 2009-2010 financial data will be reported independently during the summer of 2010 and will be incorporated into the CAA Horizontal Evaluation.

¹⁶ NRCan had received an extension of one year (2011-12) on its programs at the time of the evaluation, but this extension was a re-profiling of existing funds and did not include new resources.

¹⁷ External interviewees and training workshop evaluations pointed to the high utility of these sessions as well as their capacity to reach many members of the target population at once, saving time and effort for staff in explaining how to use the websites.

Previous page | ToC | Next page