Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2010-2011 > Evaluation Review of the Clean Air Agenda Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings
Evaluation Review of the Clean Air Agenda Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings
Previous page | ToC | Next page
The following section provides information on the purpose of and intended audiences for the Adaptation Theme Evaluation Review, the scope of the evaluation, key evaluation issues and questions, methodology and limitations.
The purpose of this document is to present a review of the available Adaptation Theme program evaluation findings, examine Theme-level issues and findings and help guide discussion on further climate change adaptation policy developments. The thematic evaluation is intended to meet the information requirements of senior management at EC, HC, INAC, NRCan and PHAC, as well Treasury Board, on the relevance and performance of the Adaptation Theme as a whole. The Thematic Evaluation also supports the Evaluation of the Clean Air Agenda.
The scope of the Adaptation Theme Evaluation Review is from 2007-2008 to the present (June 2010). Since these Adaptation programs are in the early stages of implementation (in effect, the programs are halfway through their originally planned implementation), the evaluation findings are preliminary in nature and typically focused on the outputs and immediate results that were measurable during the timeframe of the evaluations.
The focus of the review of available program evaluation findings was on relevance, outputs and early outcomes, appropriate use of resources, efficiency and economy, and design and delivery. The thematic evaluation addressed all questions required under the 2009 Treasury Board Evaluation Policy.
Relevance
As per the 2009 Treasury Board Evaluation Policy, relevance examines the extent to which the program addresses a continued need, is aligned with government priorities, and is aligned with federal roles and responsibilities. The following questions address these issues:
- Are the activities within the Adaptation Theme aligned with federal government priorities?
- Are the activities within the Adaptation Theme aligned with the priorities of participating departments?
- Are the activities within the Adaptation Theme connected with key environmental climate change and air quality needs?
Performance
As per the 2009 Treasury Board Evaluation Policy, performance examines the extent to which the program has achieved or is in the process of achieving expected outcomes and demonstrates efficiency and economy. The following questions address these issues:
- To what extent have intended outputs and early outcomes been achieved within the Adaptation Theme?
- Are there more cost-effective, economicand efficient means of achieving objectives under the Adaptation Theme?
- How could efficiency be improved under the Adaptation Theme?
- To what extent have each of the Adaptation Theme activities been implemented or are on track to being implemented as planned?
- Is the management and accountability structure for the Adaptation Theme in place and functioning as anticipated?
- What are the best practices/lessons learned from activities within the Adaptation Theme?
This section describes the two sources of evaluation evidence used to develop the Adaptation Theme Evaluation.
- Individual program evaluations – Most of the evidence presented
in the Adaptation Theme Evaluation Review is based on the available findings
from the individual program evaluations and performance data.9
The individual evaluations generated findings based on multiple lines of evidence;
therefore the evidence reviewed here already represents numerous sources of
evidence. As noted above, the individual evaluations sought to provide evidence
on the outcomes articulated in the thematic logic model to the extent possible
given the early implementation of Adaptation programs, while allowing enough
flexibility in approach to address program-specific accountability issues.
Table 2 below provides a brief
description of the status of each of the individual program evaluations at
the time of the Adaptation Theme Evaluation Review.
- Adaptation Theme Evaluation Questions – eight program managers
from participating departments were asked three additional “theme-specific”
questions. These questions, developed in consultation with evaluation representatives
from all participating departments, collected qualitative evidence to assess
the effectiveness and challenges of the Theme’s management and accountability
approach. The evidence collected from these questions was not reported in
any individual program evaluation as the questions only addressed issues related
to the Adaptation Theme as a whole.
Table 2. Evaluation Status at time of Adaptation Theme Evaluation Review (May-June 2010)
Evaluation completed and approved
Program Initiatives
Dept.
Description of Evaluation Status
1. Assistance to Northerners in Assessing Key Vulnerabilities and Opportunities
Formative Evaluation was conducted and approved by
INAC and was available for analysis in February 2010. Summative evaluation had begun and is scheduled for completion in December 2010.
2. National Air Quality Health Index and Forecast Program
Evaluation was conducted and approved by
HC and
EC in fiscal 2009-2010 and was available for analysis in March 2010.
Evaluation completed but not approved
Program Initiatives
Dept.
Description of Evaluation Status
3. Climate Change and Health Adaptation in Northern/Inuit Communities
Evaluation was conducted and awaiting approval by
HC’s Senior Management Board and was available for analysis in May 2010.
4. Improved Climate Change Scenarios
Evaluation was conducted and awaiting approval by
EC’s Departmental Evaluation Committee and was available for analysis in May 2010.
Evaluation started but not complete
Program Initiatives
Dept.
Description of Evaluation Status
5. Climate and Infectious Disease Alert and Response Systems to Protect the Health of Canadians
HC (Heat Resiliency Program) & PHAC (Pilot Infectious Disease Impact and Response Systems)
Both evaluations had begun. Preliminary findings were approved by the program and were provided to
EC Evaluation Division staff to support the Adaptation Evaluation Review in May 2010.
Evaluation not yet started
Program Initiatives
Dept.
Description of Evaluation Status
6. Innovative Risk Management Tools / 7. Regional Adaptation Action Partnerships
Evaluation scheduled for fall 2010. However, the program had conducted a National Climate Change Baseline Survey of
NRCan stakeholders, which provided preliminary baseline findings on
NRCan stakeholder awareness and understanding. The findings were available for analysis in May 2010.
The following are two limitations pertaining to the Adaptation Theme Evaluation.
- The programs under the Adaptation Theme, with the exception of the Air Quality Health Index, which was initiated under the Border Air Quality Strategy, are new programs that had been operational for roughly 18 months to two years at the time of the individual program evaluations. As noted in many of the individual evaluations, the first year of program activity was focused on early implementation activities such as staffing, and programs generally did not begin to reach external stakeholders until late 2008-2009 and early 2009-2010. As was anticipated during the evaluation planning phase, the individual evaluations generally provided evidence related to outputs and a limited assessment of early outcomes given the early stage of program implementation. The findings in the Adaptation Evaluation Review are therefore based on early assessments of program impacts; further evaluation will be required over the coming years to develop a more comprehensive picture of the impact of the adaptation programs.
- Given the short implementation period for Adaptation activities, many of
the program evaluations will be completed late in 2010-2011. The findings
in this review are, as a result, a combination of early evaluation results
and performance measurement information. Only two evaluations, the
formative Implementation Evaluation of the Assistance to Northerners in
Assessing Key Vulnerabilities and Opportunities Program (INAC), and the
Evaluation of the Air Quality Health Index Program (EC
and HC) were conducted and approved
by their respective Departmental Evaluation Committees (noted in Table 2).
The timing of approved evaluation reports was anticipated in both the Clean
Air Agenda Horizontal Evaluation Plan (in 2008) and the Clean Air Agenda Adaptation
Thematic Evaluation Plan (in 2009), given the recognition that many of the
programs would be in early implementation and the need to ensure that evaluations
were conducted after a reasonable period of implementation. Readers should
be cautioned, however, that the findings and conclusions presented here represent
the evidence available at a specific point in time (Spring 2010).
Additional discussions of evaluation-specific limitations can be found within each individual program evaluation report.
9 There were only four completed evaluations at the time of the Evaluation Review, of which only two had been approved by their respective Departmental Evaluation Committees. Evaluation Review findings also were derived from the preliminary findings of two other on-going program evaluations and a stakeholder survey used for performance measurement purposes.
Previous page | ToC | Next page
