February 2010
The Evaluation Project Team would like to express its gratitude to all those WHO contributed to this project, with special thanks to the members of the Evaluation Committee and all the interviewees WHO provided the insights and comments crucial to this evaluation.
The Evaluation Project Team was led by Gavin Lemieux under the direction of Shelley Borys, Evaluation Director at Environment Canada. The team included Linda Lee and Katheryne O'Connor from Environment Canada; Tyler Watt and Jennifer Baker from Health Canada; representatives from Health Canada's Departmental Performance Measurement and Evaluation Directorate; and, Sharla Sandrock from Government Consulting Services.
Prepared by the Evaluation Division, Audit and Evaluation Branch, Environment Canada
The National Air Quality Health Index Program (AQHI) is a program component of the Clean Air Agenda's Adaptation Theme. The AQHI is jointly managed by Health Canada and Environment Canada and was developed in order to have a nationally consistent approach to forecasting and communicating air quality information based on local conditions. It has been designed to replace the existing Air Quality Index (AQI).
While the original AQI provided a mechanism to communicate poor air conditions to Canadians, there was concern that the design did not reflect advances in monitoring and forecasting air pollution and the current state of understanding of air pollution health effects. The AQHI is based on measuring air pollutants known to harm human health: nitrogen dioxide, particulate matter(2.5µm) and ground-level ozone. The scientific foundation for the new AQHI is based on epidemiological research undertaken by Health Canada that estimates the short-term relative risks posed by a combination of common air pollutants that are known to harm human health.
The AQHI evaluation, as well as six other program evaluations from four departments, will be integrated into the Adaptation Theme Evaluation Report in summer 2010.
The evaluation addressed Treasury Board Evaluation Policy questions related to program relevance and performance. The evaluation relied on the following three methodologies to address these questions: document and data review (e.g., memoranda of understanding with provincial partners); key informant interviews with thirty seven (37) internal and external representatives; and, a review of the Program's performance measurement data.
The AQHI does appear to be a relevant federal program which addresses legitimate need for consistent air quality information and is considered a relevant federal government program by key external stakeholders. One key caveat, at present, is that the Program is still transitioning from the old AQI in key regions of the country, and the distinction between the two is not yet clear. Key highlights of findings leading to this conclusion include:
The Program generally appears to be on track in meeting its stated goals and, based on the information available, appears to be cost effective and well managed. Key highlights of findings leading to this conclusion include:
All recommendations are directed to the two responsible Assistant Deputy Ministers at Environment Canada (Meteorological Service Canada) and Health Canada (Healthy Environments and Consumer Safety Branch) in light of the irresponsibility for the overall management of the AQHI implementation. Where appropriate, the agency responsible for initiating the management action is identified to facilitate future follow-up.
(Joint EC and HC) Develop a strategy to ensure continued support to provincial partners, particularly Ontario and Quebec, to move towards full AQHI implementation. Alberta should continue to be engaged to the greatest extent possible to ensure that all provinces are at least offered the opportunity to engage in the Program.
In Quebec, an agreement amongst federal leads and Quebec partners will see the AQHI pilot expanded to Montreal, for the spring 2010. Federal support for the AQHI pilot assessment would continue until Spring 2011. The current Info-Smog Program would be recast for the spring 2011 featuring the AQHI for large urban communities and the AQI for those locations where there is inadequate monitoring. The hybrid Info-Smog Program would remain in play until such time as the Program is able to provide AQHI forecasts for those communities which are served by the AQI.
In January 2010, a modification of the Memorandum of Understanding for the AQHI forecast partnership was negotiated which would free-up Ontario Ministry of Environment staff for an AQHI location/province-wide assessment. Financial support from federal leads is budgeted to support provincial decision-making regarding implementation.
Alberta remains the only province outside the AQHI implementation planning despite recent efforts to engage staff in AQHI public meetings. The Alberta government's recent decision to introduce a revised AQI (also scaled 1 to 10) for June 2010 complicates AQHI adoption in that province. A strategy for integrating the Alberta circumstance into the national rollout will be in place in March 2011.
(EC) Develop a strategy to engage a variety of delivery channels, including local media sources, emerging media sources (e.g., social network sites, enhanced Internet delivery capabilities, push technologies) telephone access and radio. This strategy should include specific mechanisms to serve rural communities.
The prospect of Program sustainability is being enhanced through ongoing work to fold the AQHI into the dissemination pathways of the MSC. For example, a user specification document for the enhancement of AQHI on the Weatheroffice website was developed in the fall of 2009 and there have been negotiations for the implementation of enhancements over the next two years. The technology support for multiple voice products has been updated recently. This will support, for example, the provision of national AQHI forecast over the national telephone network by spring 2011.
In December 2009, Health Canada entered into a 3 year agreement with the Weather Network to fund AQHI promotion and information pieces through their various dissemination vehicles. These enhancements will be phased in over the period of the contract, with a concerted push for increasing AQHI visibility in the spring 2010.
Non-governmental partners have shown that there is a significant potential in using social media networks. In the April 2010, the best practices guide for AQHI outreach will be enhanced with a section on the use of social media networking tools, as will the Program's outreach strategy which will be completed for the summer 2010.
Develop a strategy to prioritize the engagement of at-risk populations, including the following steps: a) (HC) engage health care providers for at-risk individuals when developing partnership agreements (e.g., MOAs) with other government stakeholders and NGOs; b) (EC) in consultation with provincial partners and users, further develop the Program's approach to communicating an advisory when there are higher levels of air pollution and higher risk levels.
A national approach is being formulated to promote the AQHI with health care professionals through national professional organizations, publications and conferences. Supporting these efforts will be a HC-funded, University of British Columbia online course on the health effects of air pollution and the AQHI. This has been available since September 2009, and will run for 2 more years. In spring 2010, the Program will begin a health message review process to begin to address issues which have been raised over message efficacy by our stakeholders. A workshop in spring 2010 will kick off a multi-year process dedicated to developing and communicating more effective health messages.
In May 2009, federal leads formed an AQHI Advisory working group to examine the issue of communicating to the general public, with a focus on at-risk individuals, when there are higher levels of air pollution and higher risk levels. This working group is made up of representatives from the provinces and is helping to plan an Advisory and Special Air Quality Statement pilot project in Nova Scotia for summer 2010 which will lead to the national implementation of an advisory program.
(Joint EC and HC) Continue to refine the Program's performance measurement strategy, including: a) a revised performance measurement framework, b) analysis on the utility of the current baseline values and, c) update of the Program's current logic model.
Program principals have taken a leadership role under the Adaptation Theme with respect to logic model and indicator development. The development of a performance measurement and management framework is under way. The focus for the initial stages of the framework will be on measuring and managing performance with respect to the at-risk population, with a first draft available by June 2010.
Program principles showed significant foresight in conducting a national baseline survey in spring 2007 as well as numerous post-event surveys after smog advisories. These data are being used to establish baseline values for some of the Program performance indicators, however, more data needs to be collected, both to gather missing baseline data and to start ongoing performance measurement of indicators. Over the past few years, staff have been unable to collect this additional data because of the ongoing challenge of conducting public opinion research. This remains a clear impediment to measuring Program performance and outcomes. Qualitative measures and anecdotal evidence collected by partners provide important complementary information. These, however, in the absence of quantitative data, cannot provide a strong evidence base to support the continuous improvement objectives of the index. Also, ways to collect data that fall outside the definition of public opinion research are being pursued, and ways to fill in missing data as well as collect ongoing Program performance measurement data will continue. In addition, extensive re-analysis of existing data iscurrently being done to help establish the most accurate baseline values possible for performance indicators.
Program principals have been proactive with respect to development of a program logic model and indicators. The above-noted development of a program performance measurement and management framework will be supported by a revision in conjunction with the logic model. This will ensure that the Program logic continues to accurately reflect the Program as implementation continues, based on the most recently available information, including the results of this evaluation. A first draft will be available for EC and HC management by June 2010
a) (EC) develop a strategy to assess and improve the current AQHI observation and forecasting methodology, with input from external stakeholders, b) (HC) develop a strategy to consult with stakeholders, including academic researchers and health experts, in areas related to air quality and health, on an approach(s) to assessing AQHI-related health science issues.
A green paper will be prepared for spring 2010 for consultation and will lay out the vision for an enhanced and sustainable AQHI and Forecast program. Informed by third-party evaluation, supporting documents, ongoing input from stakeholders and experiences of staff and management, the paper will identify a number of key areas where the existing Program can be enhanced. By virtue of their importance to the sustainability of the Program, improvements underpinning the forecast production scheme and the ongoing challenges presented by air quality monitoring will be part of this document.
Assessing the health science that contributes to the formulation of the AQHI is an important piece in keeping the AQHI relevant and up to date. As the scientific formula that underpins the AQHI used the most recent epidemiological data available at the time and completed peer review in 2004, there has not been enough new data available to warrant a full re-formulation of the AQHI. That said, Health Canada is committed to keeping the index up to date. Developing a strategy to consult with stakeholders, including academic researchers and health experts, on how to best assess the impact of new health science, and how it might impact the formulation of the AQHI, is important. Health Canada will commit to developing such a strategy, with input from stakeholders, by March 2011.
(Joint EC and HC) The Program should develop a comprehensive sustainability plan to identify and address long-term issues associated with maintaining the AQHI or develop an exit strategy to ensure work to date is sustained without federal support beyond 2011.
Third party evaluations and audits have commonly pointed out that Program funding continues to be awarded on a sun-setting basis. The aforementioned green paper will provide the sustainable path forward, but in the event that future funding does not go forward as planned or funds are significantly reduced beyond what is currently available, an appropriately measured exit strategy will be included.
Environment Canada's Evaluation Division, Audit and Evaluation Branch, conducted an evaluation of the National Air Quality Health Index and Forecast Program (AQHI) with the participation of Health Canada evaluators and Government Consulting Services (GCS). This Program was selected for evaluation to support decision making, since the Program's terms and conditions expire at the end of 2010-2011. The evaluation will also be integrated into the Adaptation ThemeEvaluation Report as part of the overall Clean Air Agenda evaluationreporting strategy in the summer of 2010.
This document presents the findings and recommendations of the evaluation and is organized as follows:
The Clean Air Agenda (CAA), announced in the 2006 Speech from the Throne, is a major interdepartmental initiative with over $2.0 billion in funding over the 2007-2008 to 2010-2011 timeframe. The CAA comprises 44 programs grouped into seven themes: clean air regulations, clean energy, clean transportation, indoor air quality, adaptation, international actions, and management and accountability.1 The overall goal of the CAA is to reduce greenhouse gas emissions and air pollutants.
Nine departments and agencies are collectively responsible for the achievement of the outcomes and results at the level of the CAA and individually accountable for leading themes, managing programs and resources, and delivering and reporting on results. An evaluation plan was developed in 2007-2008 to guide the horizontal evaluation of the CAA in 2010-2011, which identified issues, questions and methodologies to be explored in each thematic evaluation. According to this plan, the CAA Horizontal Evaluation will consist of a roll-up of results from individual program/thematic evaluations of the various CAA components.
The AQHI was merged with six other programs to form the CAA Adaptation Theme. This theme includes programs at Health Canada and Environment Canada, Natural Resources Canada, Indian and Northern Affairs Canada and the Public Health Agency of Canada. Broadly speaking, the Adaption Theme programs are focused on addressing ways in which individual Canadians, organizations and various levels of government can adapt to climate change, for example by assessing the vulnerability of physical infrastructure to extreme weather events or developing the capacity to respond to extreme weather events in various regions and municipalities across the country. An evaluation plan was developed in 2009 to roll up these seven programs under one Thematic Evaluation Report, scheduled for completion in spring 2010. The Thematic Evaluation Plan, included in Annex C, outlines a thematic logic model to guide the evaluation of all seven programs and ensure a level of consistency across the theme. The current evaluation of the AQHI, however, also examines program-specific issues not included in the broader theme, particularly early outcomes which are presented in a more detailed fashion in the Program's own logic model. The current evaluation, therefore, blends and merges an examination of the Program's contribution to the broader Adaptation Theme agenda, with an assessment of early, specific, Program outcomes.
Currently in Canada, there is a shared responsibility for addressing issues arising from air pollution. This is illustrated by the following2 :
While air pollution is often associated with environmental degradation, the health impacts of air pollution have also been documented. According to the World Health Organization (WHO), for example4 :
The AQHI, based on these shared responsibilities and jointly managed by Health Canada and Environment Canada, was developed in order to have a nationally consistent approach to forecasting and communicating air quality information based on local conditions. It has been designed to replace the existing Air Quality Index (AQI). While the original AQI provided a mechanism to communicate poor air conditions to Canadians, there was concern that the design did not reflect advances in monitoring and forecasting air pollution and the current state of understanding of the air pollution health effects. For example, the AQI reported the air quality of the single worst pollutant. As a result of reporting only the single worst pollutant, there was a national patchwork of presentations with jurisdictions using different pollutants, health protection messages, and averaging time thresholds5. In addition, the AQI for fine particulates has a much lower threshold in Quebec than in Ontario. Therefore, for the same actual PM2.5ambient concentration, the AQI could be poor in Quebec but only moderate in Ontario.
The index rating for the AQHI is the sum of the health risks from each of the pollutants in the index.6It is an indicator of the short-term health risks associated with air quality, based on measuring air pollutants known to harm human health. It takes into account the effect on health of even low levels of exposure to multiple pollutants, such as ground-level ozone and other components of smog. The health risk is calculated based on a combined exposure to nitrogen dioxide, particulate matter (2.5µm) and Ground-level Ozone.
The scientific foundation for the new AQHI is based on epidemiological research undertaken by Health Canada that estimates the short-term relative risks posed by a combination of common air pollutants that are known to harm human health. The new AQHI focuses on pollutants that can be measured and provides health messages that involve actions that individuals and caretakers can take to limit short-term exposure to air pollution.
Table 2.1 shows the AQHI (for Toronto). This example includes both a numeric valuefor the air quality, and health information based on the level ofrisk associated with the air quality.
The historical basis of air pollution monitoring and the original AQI is based on the Index of the Quality of the Air (IQUA) which was first introduced in 1979 to report on a set of national voluntary goals for indoor air quality called the National Ambient Air Quality Objectives (NAAQOs). The National Air Pollution Surveillance (NAPS) Network was established to support the collection of national air pollutant data. NAPS is a partnership agreement between the federal government (Environment Canada) and the provinces and territories to collect data from roughly 300 monitoring stations mainly in large urban areas.
In 2001, Environment Canada began to reassess the AQI based on concerns expressed by stakeholders, such as provincial and municipal health departments and non-governmental organizations, that the air quality index in place was out of date. The existing AQI reported daily maximum values on a 0 to 100 point scale. If, for example, ozone was the highest reported daily value, that single pollutant was reported. The 100 point scale included a threshold value to alert individuals to modify behaviour. It did not, however, contain any information specific to the health risks associated with the pollutant, nor did it contain any information on mitigation strategies that may be undertaken by individual Canadians or, specifically, by "at-risk" groups that had existing respiratory illnesses and conditions.
Further assessment of the AQI by Environment Canada and Health Canada revealed three main areas of concern:
The current AQHI is based on federal consultations with a variety of stakeholders, including, but not limited to, the provincial and territorial governments. In 2001, Environment Canada and Health Canada developed a management committee and three working group committees (Health Aspects, Monitoring and Data Analysis, Market Research and Marketing) to oversee the development of the revised AQHI.
In the first phase of development, the Monitoring and Data Analysis group developed a set of technical recommendations for the AQHI. The following points were noted in the report:
Health Canada scientists conducted further assessments on the impact of air pollution on human health and concluded that while there was no definitive answer on developing multi-pollutant health indices, the proposed AQHI was consistent with World Health Organization (WHO) guidelines for monitoring air quality.
Environment Canada and Health Canada also conducted various additional tests and national workshops between 2002 and 2005 to share information on the progress of the new index and obtain input into its development. For example, in 2004, the two departments conducted an "AQI Health Message Development Workshop" in Ottawa.8 The workshop addressed messaging for the general population, sensitive populations based on existing medical conditions (e.g.,asthma) and sensitive populations based on age (e.g., seniors).
During this period, the federal government created the Border Air Quality Strategy (BAQS), which was resourced from 2003 to 2007, to engage the United States and provinces in addressing the goals of the CAA. The activities under the BAQS helped support the development of the AQHI through supporting Environment Canada's development of an air quality forecast methodology and promoting the use of air quality forecasts. Health Canada was also involved in additional analysis of the feasibility of a multi-pollutant air quality system and in engaging key agencies and high risk groups in developing a messaging approach to air quality forecasts.
Finally, pilot tests of the AQHI were conducted in British Columbia (2005-2007), Nova Scotia (2006) and Toronto (2007). Feedback from all three pilot tests were positive and pointed to the importance of reaching individual Canadians through a variety of media sources, such as television, radio and print media.
The current evaluation examines the development of the AQHI since FY 2007-2008. As the Program is currently engaged in ongoing implementation activities, the evaluation examines the Program's progress towards meeting the short-term goals laid out in its logic model, as well as a preliminary assessment of the Program's ability to impact public awareness and behavioural change.
The following are the objectives of the AQHI:
The reach of the AQHI extends to a broad range of stakeholders and beneficiaries that include10:
The following resources and funding amounts were allocated to the Program over a period of four years starting in 2007-2008:
| 2007-2008 ($m) | 2008-2009 ($m) | 2009-2010 ($m) | 2010-2011 ($m) | Totals ($m) | |
|---|---|---|---|---|---|
| Environment Canada | 4.5 | 5.2 | 5.8 | 5.8 | 21.3 |
| EC Grant/Contribution Component | 0.1500 | 0.440 | 0.250 | 0.250 | 1.090 |
| Health Canada | 1.5 | 2.8 | 2.2 | 2.2 | 8.7 |
| HC Grant/Contribution Component | nil | nil | nil | nil | nil |
| Total | 6.0 | 8.0 | 8.0 | 8.0 | 30.0 |
Ananalysis of budgeted versus expended (for the first two complete fiscal years) is presented in the results section.
The Program's logic model illustrates its activities, outputs, target audiences, direct outcomes, as well as intermediate and final outcomes. As mentioned previously, the evaluation of the AQHI is occurring in the context of a larger set of evaluations under the Adaptation Theme of the Clean Air Agenda (discussed in the following section). As a result, the evaluation examined early outcomes of the AQHI as defined in the Program's logic model, but also integrated larger, thematic outcomes as defined in the Adaptation Theme Logic Model, (located in Annex A).
Two previous reports should be highlighted. The first, as mentioned above, is the Border Air Quality Strategy (BAQS) Evaluation conducted by Environment Canada in 2007. The purpose of the BAQS Evaluation was to assess and determine Environment Canada's outputs, achievements, and anticipated results related to their responsibilities for Air Quality Forecastin under the four-year Border Air Quality Strategy (BAQS) which ran from 2003-2004 to 2006-2007
The evaluation made five recommendations, listed below:
The management actions associated with recommendations three and five are examined as part of the AQHI evaluation as these management actions had not yet been completed during the planning phase of the AQHI (the other three had been completed). The management response to these recommendations, as well as the evidence provided to assess progress on these actions, are included in section 4.0 of this report.
Also worth noting is a second report, the 2009 Commissioner of the Environment and Sustainable Development (CESD) audit of the AQHI. The CESD commissioned the audit to examine progress made on the government's commitments to develop a Canada-wide air quality index based on health risks. The general conclusions of the audit were:
The audit, conducted only six months prior to the data collection for this evaluation, is used as a source of evaluation evidence where pertinent.
The following sections outline the evaluation purpose and scope and the data collection approach and methods used.
The following section details the manner in which the evaluation addresses current Treasury Board Evaluation Policy questions, evaluation methods and limitations.
As per the 2009 Treasury Board Evaluation Policy, the issue area of relevance examines the extent to which the Program addresses a continued need, is aligned with government priorities, and is aligned with federal roles and responsibilities. Specific evaluation questions in the area of "relevance" that were examined using multiple lines of evidence were as follows:
As per the 2009 Treasury Board Evaluation Policy, the issue area of performance examines the extent to which the Program has achieved or is in the process of achieving expected outcomes, and demonstrates efficiency and economy. The outcomes examined here include both Program outcomes as defined in the Program's logic model as well as outcomes identified in the Adaptation Theme Logic Model related to the AQHI. The specific evaluation questions that were examined using multiple lines of evidence were as follows:
This section describes the methods that were used to conduct the evaluation of the AQHI as well as limitations of the evaluation.
Document and Data Review – The evaluation team reviewed key documents including relevant legislation, academic research and evaluations of similar programs, Program work plans and budgets. A total of 51 documents were analyzed during the course of the evaluation. A complete list of documents reviewed is presented in Annex B.
Key Informant Interviews – Environment Canada subcontracted a portion of the key informant interviews to Government Consulting Services (GCS). GCS conducted 28 interviews with external stakeholders. Interviews with nine additional program managers and staff were conducted by EC and HC evaluation representatives. A semi-structured interview guide was developed by GCS and evaluation division representatives from both departments, and reviewed by members of the Evaluation Steering Committee.
A preliminary sample of 44 potential interviewees was drawn up in consultation with members of the Evaluation Steering Committee. Regional representation was considered critical given the varied regional progress and implementation issues of the AQHI. Effort was made to ensure that there was representation from the main groups directly involved in Program implementation, regional Program representatives, NGOs and government partners. Table 3.1 provides the regional distribution of respondents.
| Region | Totals |
|---|---|
| External Stakeholders | |
| British Columbia | 3 |
| Prairies | 3 |
| Ontario | 7 |
| Quebec | 3 |
| Maritimes | 6 |
| Others | |
| National/International | 6 |
| Program Management (including members of Evaluation Steering Committee) | 9 |
| Total Number of Interviewees | 37 |
Interview notes were prepared and entered into a spreadsheet to facilitate analysis. A thematic analysis of the interview information was then completed. The interview response analysis grid shown in Annex C was used to analyze the interview responses.
Interview findings are generally categorized as a "majority", indicating 19 or more respondents, or a "minority" indicating 18 or less respondents. Regional findings are provided in those cases where there were distinct regional differences or where regional information was pertinent to addressing the evaluation question.
Program Performance Measurement Data – The Program undertook several steps to develop the performance data used in this report. An Air Quality Forecast Program (AQFP) baseline survey was administered in 2007 to over 4000 individuals, and 10 post-smog event surveys were administered to a minimum of 400 individuals in various locations across Canada after a smog advisory had been issued. The Program also developed a performance measurement grid, based on the logic model, to guide performance reporting. The Program used the services of an independent contractor to report the data by performance indicator. This information was made available to the evaluation team during the conduct of the evaluation. Evaluation team members also conducted an independent analysis of the survey data to better understand thelimits and interpretability of these data sources.
The following six limitations to the evaluation methodology are noted:
The Program evaluation took place at roughly the midpoint of the Program's implementation. While evaluations typically examine a program's performance about one year prior to completion of the funding, the need to merge the findings from this evaluation into the broader Adaptation Theme evaluation report, and the subsequent merger of all evaluation data into a summary Clean Air Agenda evaluation for fall 2010, necessitated an earlier evaluation. As a result, while the evaluation reports on the achievement of outcomes, these findings are fundamentally preliminary in nature and may not reflect the final achievement of Program outcomes at the end of the funding period.
The provincial/regional progress reports, required of funded external partners, are based on different time periods as not all partners had completed progress reports at the time of the evaluation, or were up to date on their reporting. As a result, it was difficult to obtain a single national profile of the AQHI at the time of the evaluation.
Although the evaluation team sought perspectives from each of three stakeholder groups (program representatives, NGOs, and government partners) in every region, findings are limited by the small number of interviewees in each region. Because of this, it was difficult to draw consensus by region. Thus, findings by region have been presented using the actual number of interviewees that provided a response in order to illustrate their materiality.
Interview findings are based on the perspectives of AQHI delivery partners (Program representatives, NGOs, and government partners) as there were limited resources to gather data on target audiences. When asked about behavioural changes in other target populations such as at-risk groups, interviewees were therefore only able to provide their perception of what has occurred within the target group. Thus, responses are based on the perceptions of delivery partners rather than the actual target population.
There are a variety of limitations associated with the Program performance data available in the final Performance Indicator Baseline Report. One major limitation, however, is the fact that all the surveys, either the 2007 baseline survey or any of the post smog event surveys, asked questions related to the AQI. There are no current surveys which examine performance issues related to the AQHI. While the AQI was considered a temporary proxy measure of the AQHI, given that it was also an air quality forecast measure, none of the baseline values noted in this report refer to either the health information integrated into the AQHI; nor can any baseline data be related to the specific activities funded under the AQHI.
There is limited comparability of this program to similar programs in other jurisdictions. Other countries, most notably the United States, do have an air quality forecast system. However, they are notably different given that they are in place to monitor air pollution as part of a regulatory framework, do not operate in a comparable federal environment and do not focus on health-related messaging. As a result, potential comparisons across issues such as impacts of cost-effectiveness were limited.
The findings of this evaluation are presented below according to evaluation issue (relevance and performance) and related evaluation questions. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.
A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings indicate that:
A summary of ratings for the evaluation issues and questions is presented in Annex D.
Except where specifically mentioned, no notable differences were found in findings pertaining to Health Canada or Environment Canada practices and processes. Unless otherwise specified, interview responses were common across both departments.
The following section details thefindings by evaluation question.
The AQHI does appear to be a relevant federal program which addresses legitimate need for consistent air quality information and is considered a relevant federal government program by key external stakeholders. One key caveat, at present, is that the Program is still transitioning from the existing AQI in key regions of the country, and the distinction between the two is not yet clear.
Evaluation Issue: Relevance
1. Are the activities within the AQHI connected to key air quality information needs?
Indicator(s)
External opinions on key air quality information needs
The extent to which stakeholders believe the AQHI is addressing key air quality information needs
Evidence from other sources (e.g., CESD Audit) indicating that the Program is connected to key air quality information needs
Rating
Achieved
The Program appears to be addressing key air quality information needs by informing the public of the potential health impacts of air quality and by addressing earlier concerns about the usefulness of air quality information under the AQI.
The AQHI appears to address issues linking air quality with health. Numerous independent peer reviewed studies11 in the United States, Canada and Europe have found a link between air pollution and health. For example:
Exposure to air pollutants such as airborne particulate matter and ozone has been associated with increases in mortality and hospital admissions due to respiratory and cardiovascular disease.12
The Canadian Medical Association also notes:
These health outcomes were identified as loss of productivity, increased health care costs, and reduced quality of life.
There is evidence that the Canadian public, particularly those living in urban areas, are concerned about the impacts of air quality on health. In post smog event surveys, the majority of urban respondents (e.g., in Windsor, Toronto and Montreal) indicated that air pollution presented either a "very serious" or "somewhat serious" hazard to human health. Residents of Windsor, in particular, appear to be concerned about air quality: in a 2008 survey, almost half (48%) of respondents indicated that air pollution was a "very serious" health hazard. Respondents from relatively rural areas (e.g., Abitibi, Mauricie, the Annapolis Valley in Nova Scotia) were less likely to view air pollution as a serious health hazard.
The 2009 CESD Audit indicated that, as early as 2001, there were concerns from a variety of stakeholders that the then existing Air Quality Index (AQI) did not adequately link air quality information with information on health and the health risks associated with air quality. Furthermore, the CESD audit indicated that the AQHI, with its focus on linking air quality information with health information, represented a progressive step towards addressing these concerns.
The majority of interviewees indicated that the AQHI successfully creates an information link between air quality and health; and that the AQHI addresses the requirement for uniform air quality reporting through a nationally standardized program.
In terms of the extent to which the AQHI addresses issues in disseminating air quality information, interviewees noted that the AQHI addresses issues of awareness by informing the public of air quality information through their website and other media sources such as the Weather Channel. However, with reference to issues regarding the dissemination of air quality information in Canada, the majority of interviewees suggested that there is a general lack of awareness among Canadians regarding air quality and the relationship between air quality and health.
Survey research14 conducted by the Program indicates that the majority of Canadians are not necessarily aware of air quality forecast information in their region. For example, surveys conducted in 2007 indicated that 35% of Canadians within AQI forecast regions were aware of air quality forecasts.15 This survey was conducted prior to the implementation of the current AQHI and therefore these results should be treated with caution. It is worth noting, however, that a key component of the AQHI's relevance is to inform Canadians on the link between air quality information and health messages. The potential impact of the AQHI may be diminished if Canadians are unfamiliar with the AQHI.
Evaluation Issue: Relevance
2. Are the activities within the AQHI aligned with federal government priorities?
Indicator
Demonstration of alignment between Program objectives and federal government priorities
Rating
Achieved
The AQHI is aligned with current federal government priorities.
All the programs under the Clean Air Agenda represent components of the federal government's commitment to mitigating greenhouse gas emissions and air contaminants. The AQHI is therefore aligned with federal government priorities through its inclusion in this large, interdepartmental initiative. Specifically, the AQHI is one of the seven programs under the Clean Air Agenda Adaptation Theme and as such is aligned with current federal government policies on adaptation.
The federal government is currently working with a tripartite group comprised of industry, NGOs and provincial governments to develop alternative strategies to its 2007 plan based on Turning the Corner. The draft proposal developed in February 2009, entitled the Comprehensive Air Management System (CAMS), suggests the development of a comprehensive Canadian air management system.16 It states that:
The initial focus on this system is on fine particulate matter (PM2.5)and ground level ozone and their precursor gases […] It is recommended that these substances be addressed through a similar multi-stakeholder process, and/or be considered as a related component of this system in the future.
It is important to note that the AQHI was not developed as an air quality management tool; however the AQHI does appear to be broadly linked to the federal government's evolving approach to managing air quality and regulatory development.
Evaluation Issue: Relevance
3. Are there areas of duplication and/or alignment among AQHI related activities between the federal government, provincial governments, OGDs and NGO/private sector stakeholders?
Indicator
The extent to which stakeholders indicate that there is duplication and/or alignment between the AQHI and related programs
Evidence regarding the impact on air quality information if the AQHI was not in place
Program data indicating efforts to align with provincial programs
Documents indicating alignment/duplication between AQHI and related programs
Rating
Some progress/ attention needed
As indicated in the overview of the Program's development, the AQHI operates in an environment of complex federal/provincial jurisdiction, both in terms of sharing air quality and forecasting data among multiple partners and in terms of the provision of health care. Within this complex operating environment, there is evidence that the Program is taking steps to ensure alignment with provincial partners. There is concern, however, that the AQHI is still viewed as a duplication of the AQI, not an improved approach, in key regions of the country.
In terms of alignment of the AQHI to current provincial government priorities, provincial stakeholders reported that the provinces aim to promote air quality through provincial air quality management plans. Through education and awareness, these plans aim to minimize the risk to public health from air pollution and to promote health activities that directly respond to issues on air quality. For provincial stakeholders, therefore, the AQHI complements the priorities of provinces by serving as a primary communicator of health risks associated with air quality.
Interviewees in British Columbia, the Maritimes and the Prairies indicated that there is an alignment of the AQHI to current provincial government priorities. In Quebec, interviewees expressed concern that the priorities cannot align due to continued wide usage of AQI and Info-Smog Quebec in most areas of the province (with the exception of Quebec City and Gatineau).
In reference to duplication and alignment of AQHI-related activities to other similar programs, the majority of interviewees made reference to the AQI as a primary area of duplication. This was expressed by the majority of interviewees in British Columbia and in Ontario. In Quebec, interviewees noted that the AQI and Info-Smog Quebec duplicate AQHI. In the Prairies, interviewees responded that there is no duplication with other indices in Manitoba and Saskatchewan, but made reference to Alberta's desire to maintain the AQI as their primary index.
In the Maritimes, interviewees noted a developing alignment between AQHI and NGOs, for example, the complementary activities of organizations such as the New Brunswick Lung Association, Saint John Coalition for Air, and Clean Air Nova Scotia.
An analysis of the provincial memoranda of agreements (MOA) points to an effort to align with existing provincial programs.17 For example:
The analysis, however, also points to the somewhat different levels of implementation between provinces. British Columbia for example, appears to be more advanced in implementing health messaging programs than the rest of the country in terms of the number and variety of outreach tools used in that region. The relatively higher level of implementation in British Columbia was noted independently by senior program managers at both Health Canada and Environment Canada and was seen as a result of strong provincial leadership and the fact that the province had been an early adopter by participating in an early AQHI pilot. Current implementation differences between provinces may also be understood as a function of the early timing of the evaluation as provinces and/or regions that implemented the AQHI earlier are somewhat farther ahead than other regions of the country. The timing of the evaluation limited the extent to which the evaluation could assess whether and to what extent any structural or policy issues in provinces impacted the successful implementation of the AQHI.
The CESD Audit found that positive relationships have been built between stakeholders and AQHI. The report noted:
However, the same report noted that, in spite of concerted efforts to work inunison with external stakeholders, "working with the provinceson issues related to the total or partial phase-out of the existingair quality indices" remains a key short-term challenge for theAQHI. 19
As a measure to further establish the continuing relevance of AQHI, interviewees were asked to identify what gaps might occur if AQHI did not exist. Many stated that there would be decreased awareness of the direct health link to the reported level of air quality and some reported that there would be an overall lack of information on air quality. Seven of the external stakeholders noted, however, that individuals would still be able to rely on the AQI (or provincial equivalent) in the absence of the AQHI, but would not have the same exposure to health-related information.
The Program generally appears to be on track in meeting its stated goals and, given the information available, appears to be a cost effective and well managed program.
Evaluation Issue: Performance
4a1. Sharing real-time provincial data
Indicator
The extent to which stakeholders indicate real-time provincial data are being shared
Documents indicating existence of real-time data sharing
Rating
Some progress/ attention needed
One key measure of the Program's success is the extent to which the system is in place to ensure ongoing data sharing between provincial monitoring stations and the federal government. Program management commented on the complex nature of this system in that the AQHI relies on a network of monitoring stations which are under provincial jurisdiction. Generally, the system for sharing data was seen to be in place but ongoing challenges include ensuring a consistent national approach and working with a complex array of provincial partners.
The CESD Audit noted that the current NAPS network is in place to provide data sharing between the provinces and the federal government. Program management also indicated satisfaction with the current data sharing system, noting that data management systems were in place in most provinces to ensuring ongoing data sharing.
The majority of external stakeholders indicated that the AQHI is working well towards achieving its objective of improved sharing of real-time provincial air quality data. Some of these stakeholders noted that the provinces share data readily with other provinces and/or with the federal government partners. The following are examples of other comments made:
One interviewee noted that the sharing of data is mainly attributable to the data being made publicly available on weather sites.
One interviewee indicated that foundational work for information sharing was completed prior to the implementation of the AQHI and thus no further improvement in sharing is necessary. The noted exception to this was Alberta.
In some instances, the lack of monitoring stations was seen as a barrier to further data sharing by both external stakeholders and Program management. For example, in the Maritimes, interviewees indicated that sharing was happening "to a limited extent," noting that in New Brunswick, the AQHI has not expanded outside of Saint John. This was due to the fact that the implementation of AQHI in Saint John was a pilot project and that there is a lack of monitoring stations in other areas of the province.20
In the Quebec Region, interviewees indicated that an improvement in sharing of real-time data has not occurred. In this case, it was felt that it is too early to assess this outcome since the AQHI is only beginning to be implemented in this province.
Program management indicated that there were ongoing challenges.
There is no regulation that would force provinces to share data; the AQHI relies heavily on consensus building between partners and the voluntary participation of provinces. The resulting process can be time consuming.
A related issue is the fact that many provinces have, historically, not reported air pollutants, such as particulate matter, at the rate required for the AQHI, which relies on continuous real-time monitoring to produce the three-hour moving average of values.
Certain provinces have indicated a desire to include other pollutants in the AQHI, particularly those that produce noticeable smells in the air (though not necessarily dangerous health hazards). The Program is currently engaging these provinces to find suitable solutions.
Interviews with Program management and a review of Program documents21 indicated that the Program is continuing to improve the data sharing system, for example:
creating a formal annex to the current NAPS agreements to formalize the data sharing required for the AQHI;
developing a standard operating procedure to advise forecasters on making amendments to the daily AQHI forecast; and
time stamping data coming from provincial monitoring stations to verify the issue time for data.
Evaluation Issue: Performance
4a2. Transition to AQHI observations and forecasts
Indicator
Number of census metropolitan areas covered by AQHI
Extent to which stakeholders indicate that the Program is progressing towards transition to AQHI
Challenges/barriers to AQHI transition
Rating
Some progress/ attention needed
The evidence points to a generally successful transition from the AQI to the AQHI in many parts of Canada. However, there remain several key areas at risk (e.g., Quebec, Alberta) and other regions which are transitioning slowly, such as parts of Ontario.
The original target set for implementation of the AQHI in 27 Census Metropolitan Regions (CMAs) 22 was the end of FY 2010-2011.23 The Program modified its intended approach during the first year of implementation, at the request of provincial partners, to ensure that its activities were aligned with provincial jurisdictions. As a result, the implementation of the AQHI is not limited strictly to CMAs, but additional, smaller, communities. The data in Table 4.1 indicate that the AQHI is currently available in 27 municipalities (covering 13 million individuals), has initiated work in another 6 municipalities (covering 4.0 million individuals) and may be implemented in an additional 14 municipalities in FY 2010-2011 (covering 3.3 million individuals). A risk assessment conducted prior to the implementation of the AQHI indicated that readiness to implement across all 27 CMAs was considered a risk. As a result, the Program developed alternative "contingency plans" to implement in smaller communities. Program management cited the risk assessment process as a factor in the Program's ability to modify its original approach.
If the Program is able to maintain its proposed implementation schedule, it will meet its original projections of CMA coverage as well as expand coverage to smaller communities. While the AQHI has been implemented in several key regions of the country, including Toronto and Vancouver, major industrial sectors in Canada, such as Alberta and the Quebec City to Windsor corridor, are either pending or are being implemented slowly. Finally, it should be noted that the definition of "implementation" is, in this case, limited strictly to the availability of a local AQHI forecast and does not necessarily indicate a full range of potential outreach or health promotion activities.
| Municipality | Population (M) | Year Available | |||
|---|---|---|---|---|---|
| FY 07-08 | FY 08-09 | FY 09-10 | FY 10-11 | ||
| AQHI Available (CMA) | |||||
| Vancouver | 2.285 | • | |||
| Victoria | 0.337 | • | |||
| Kelowna | 0.162 | • | |||
| Abbotsford | 0.165 | • | |||
| Toronto | 5.509 | • | |||
| Ottawa/Gatineau | 1.168 | • | |||
| Quebec City | 0.738 | • | |||
| Halifax | 0.386 | • | |||
| Oshawa | 0.348 | • | |||
| Saint John | 0.126 | • | |||
| Winnipeg | 0.712 | • | |||
| Saskatoon | 0.241 | • | |||
| Regina | 0.201 | • | |||
| St John's | 0.184 | • | |||
| PEI province | 0.139 | • | |||
| Cape Breton (includes Sydney) | 0.109 | • | |||
| AQHI Available (Smaller Municipalities) | |||||
| Kamloops | 0.089 | • | |||
| Vernon | 0.051 | • | |||
| Nanaimo | 0.085 | • | |||
| Prince George | 0.085 | • | |||
| Quesnel | 0.024 | • | |||
| Corner Brook | 0.027 | • | |||
| Kentville | 0.026 | • | |||
| Williams Lake | 0.010 | • | |||
| Whistler | 0.009 | • | |||
| Total Population (AQHI Available) | 13.216 | ||||
| Initiated | |||||
| Montreal | 3.695 | • | |||
| Moncton | 0.118 | • | |||
| Fredericton | 0.081 | • | |||
| Brandon | 0.041 | • | |||
| Prince Albert | 0.041 | • | |||
| Pictou | 0.036 | • | |||
| Port Hawkesbury | 0.004 | • | |||
| Duncan | 0.020 | • | |||
| Total Initiated | 4.016 | ||||
| Population proposed for possible 2010-2011 implementation | |||||
| Hamilton | 0.720 | • | |||
| London | 0.468 | • | |||
| Kitchener | 0.468 | • | |||
| St.Catharines | 0.396 | • | |||
| Windsor | 0.331 | • | |||
| Sudbury | 0.163 | • | |||
| Kingston | 0.155 | • | |||
| Thunder Bay | 0.124 | • | |||
| Swift Current | 0.016 | • | |||
| Castlelgar | 0.008 | • | |||
| Cranbrook | 0.005 | • | |||
| Total Proposed for 2010-2011 | 2.854 | ||||
| Population Pending25 | |||||
| Calgary | 1.139 | ||||
| Edmonton | 1.081 | ||||
| Sherbrooke | 0.167 | ||||
| Saguenay | 0.152 | ||||
| Trois-Rivieres | 0.144 | ||||
| Total Pending | 2.683 | ||||
Further review of Program work plans indicated that AQHI management does recognize the risks posed to limited transition in key areas. The current work plan identifies the following risks to the successful transition to the AQHI.26:
Regional Differences: Different regions have different concerns and priorities (such as an interest in giving more or less weight to different pollutants in the index to reflect local issues such as odor or visual cues like smoke) resulting in some resistance to the national approach of the AQHI. Other concerns raised have been with respect to rural applicability of the AQHI and concerns about sites that do not have a full suite of monitors.
Jurisdictional: Ontario, Quebec and Alberta are the three jurisdictions of concern with respect to the implementation of the AQHI:
Ontario has not identified a lead department for the AQHI and thereis currently no champion in place to bring the AQHI through their bureaucratic process;
Quebec has identified only three cities that will have the AQHI, andthey are currently introducing the AQHI in the context of a pilot.There is an underlying issue of harmonizing the AQHI with their existing InfoSmog Program; and,
Alberta Environment has been reticent to participate in the process since its inception.
External interviewees highlighted and validated many of the risks identified by the Program. The majority of interviewees noted that the AQHI has been successful in making the transition from current AQI observations and forecasts to AQHI observations and forecasts. Of this group, the majority noted that the AQHI now has presence in most provinces. However, a small number of these interviewees noted that two provinces, Alberta and Quebec, are currently not participating in the AQHI Program as anticipated. Alberta was not participating in the AQHI due to differences in the scientific approach underpinning the formulation and Quebec was noted to still be in the transition to building the AQHI into its programming.
In Ontario, interviewees noted that the AQHI is still in transition, pointing out that some locations such as Windsor, Sudbury and North Bay currently do not have AQHI, and that it has been problematic to expand beyond the Ottawa area and Toronto. Interviewees noted that one factor contributing to the partial provincial transition is that, in some areas, only two out of the mandatory three pieces of monitoring equipment are in place, slowing the implementation of the AQHI.
The majority of interviewees made positive comments about how collaborative arrangements between the AQHI and the provinces/municipalities have helped to facilitate the transition to the AQHI. These interviewees noted that existing collaborative relationships assisted in achieving the transition to AQHI observations and forecasts. Some of those WHO commented noted that communication between AQHI partners has been very productive, noting specifically that the ability of the AQHI team to engage in direct discussions with a variety of jurisdictions, including sub-provincial ones, has been a catalyst to rapid uptake of AQHI information.
Program management indicated that the move to working with provinces, as opposed to strictly working with CMAs, has been advantageous as it has allowed for greater potential reach into rural areas without necessarily compromising the number of CMAs engaged in the AQHI.
The CESD Audit also noted that the full national implementation of the AQHI was an area of concern, particularly in Alberta.
Evaluation Issue: Performance
4a3. Public Availability of AQHI products and services
Indicator
Number/examples of AQHI products and services
Extent to which stakeholders indicate that the Program is making AQHI products and services available to the public
Challenges/barriers to availability of AQHI products and services
Rating
Some progress/ attention needed
The AQHI has developed an approach to creating publicly-available AQHI products and services through such mechanisms as memoranda of agreement (MOAs) with the provinces; however, improved use of local media and other alternative sources of media were seen as areas for improvement.
The majority of external stakeholders interviewed indicated that since the implementation of the AQHI in 2007, the Program has increased public availability of AQHI products and services. The two primary ways in which this increase has been realized has been through the Internet and the Weather Network. A few interviewees noted that AQHI information can be found on provincial and federal websites. A few interviewees also noted that providing AQHI information on the Weather Network creates the potential for AQHI products and services to be completely available to the Canadian public. Interviewees from the Prairies and Quebec indicated that public availability was an ongoing challenge as a result of the lack of AQHI adoption in Alberta27 and the early stages of implementation for Quebec, respectively.28
Challenges to increased public availability were also noted by interviewees. A few interviewees noted that there is still some confusion regarding the differences between the AQI and AQHI. One interviewee noted that there are regional restrictions on the AQHI data that are hampering public availability of AQHI information. For example, two interviewees noted that AQHI information is only available within certain regions (Saint John versus other parts of New Brunswick, and Halifax, Sydney and Annapolis Valley versus broader Nova Scotia). This was true in Ontario as well, where interviewees cited the success of the AQHI's availability through broadcast media in Toronto and perhaps the Greater Toronto Area but not in other locations in the province; they further stated that the continued reliance on the AQI is confusing for media and viewers. It was also noted that an arrangement has been developed with the Weather Network to deliver AQHI information, however, the extent to which the Weather Network reports AQHI across the country varies by region.
A few interviewees suggested that partnerships with media were an area for improvement. Two of these interviewees noted that local media channels have yet to be engaged.
Many interviewees thought that partnerships between AQHI and broadcast media contribute to the increased availability of AQHI products and services. The formal arrangement with Weather Network was seen as a particularly important contributor to this outcome since most print media use the Weather Network as their main source of weather information. The further engagement of the Weather Network was viewed as important because of its audience of seven million homes through cable subscriptions in Canada. A few interviewees noted that relationships with local media are currently limited but their development may serve as an opportunity for further improvement of this outcome.
Federal AQHI website(s), including both airhealth.ca and the main Weatheroffice website, are in place and contain information on AQHI values for selected regions as well as health-related information on actions that can be taken to mitigate the risks associated with poor air quality.
Evidence from provincial reports indicated that the MOAs appear to have been successful in generating regional AQHI products. The following examples illustrate these products.29
In British Columbia:
Mailed out promotional material to mayors and council for 9 municipalities including:
The following print tools
10 000 tri-fold brochures
5500 rack cards
100 posters
speaking points
frequently asked questions
backgrounders
surveys
banners
signage;
The following promotional give-aways
4000 recycled reusable shopping bags
500 promotional energy bars
400 fridge magnets
stickers
20 T-shirts.
In New Brunswick:
Outreach and presentations given to 29 key stakeholder groups; and
Delivery to selected target audiences of printed communication materials such as:
202 AQHI posters
507 fact sheets
124 units (50 pages each)AQHI tear-pads
350 AQHI index cards with magnets
90 letters to physicians
6 newsletters.
The majority of interviewees provided suggestions for other media that could be engaged to further increase public availability of AQHI products and services. One of the more common suggestions included the engagement of local newspapers since these are well-read sources of information that have not yet been sufficiently engaged and could provide a good means of disseminating AQHI information on a daily basis along with the weather forecast. Other suggestions were to provide AQHI information through radio, Blackberry messaging, and social networking sites such as Twitter and FaceBook, as well as using "push" technologies30 such as Instant Messaging.
Program management also noted that they were beginning to explore approaches to using social network sites, given the increased use of that form of media.
Evaluation Issue: Performance
4a4. Dissemination of materials and/or advice on air quality and health
Indicator
Number/examples of dissemination of AQHI materials and/or advice on air quality and health
Extent to which stakeholders indicate that the Program is disseminating materials and/or advice on air quality and health
Challenges/barriers to dissemination
Rating
Some progress/ attention needed
While the AQHI is taking steps to disseminate materials with health advice related to air quality, more could be done to reach targeted populations such as seniors. Barriers included a general lack of awareness outside those directly involved in the regional implementation of the AQHI.
Many interviewees agreed that the AQHI is making progress towards disseminating materials and/or advice on air quality and health. The majority of interviewees indicated that the AQHI was progressing on this outcome to a "great extent". Of these respondents, some noted that the media has played a large role in the dissemination of AQHI materials. The majority of these interviewees also noted that the Web is the most prominent media form that assists with the dissemination of materials. It was noted that, federally, EC has made great efforts to post AQHI information.
The majority of interviewees from Quebec noted that the AQHI was only progressing "to a limited extent" in this area. This is possibly because AQHI has not yet been established in Quebec, with only pilot projects being completed to date. As well, interviewees noted that AQHI has not yet been established on Montreal Island.
Interviewees whose organizations were directly involved in distributing AQHI materials mentioned brochures, fridge magnets, coasters, posters and note pads that describe the use of the AQHI index. Other types of dissemination activities that were mentioned by interviewees included: presentations to community organizations and community workshops; ongoing education of broadcast media; and development of educational resources for future distribution in schools.
Interviewees provided suggestions for improvement in this area. Half of these suggestions came from Ontario where dissemination efforts seem to have focused on the city of Toronto with some efforts beginning to be placed on the Greater Toronto Area. Interviewees suggested that there needs to be an "advisory" associated with the AQHI to better highlight the days with particularly high ratings of AQHI, noting that AQHI needs to be more active in their information dissemination, rather then depending heavily on posting the information on a website that people have to go to and get the information themselves. A small number of interviewees highlighted the gap in disseminating information to seniors. Their suggestion was for the AQHI to disseminate information in a more targeted fashion to include pamphlets, posters and presentations to seniors since they may be less likely to use the Internet as a source of information. One interviewee estimated that more than 70 per cent of seniors have not yet been reached.
The majority of interviewees suggested additional partnerships that should be fostered in order to further meet AQHI objectives. A few of these interviewees noted that partnerships with education ministries should be developed. One of these interviewees noted that while ministries of education have started to be engaged, there is a further need to ensure that AQHI is incorporated into school curricula. Another interviewee noted that an outdoor air quality program in schools is anticipated and the AQHI should be a part of this initiative.
Table 4.2 indicates one potential problem regarding the dissemination of the AQHI. The table presents data from two post-smog event surveys in 2007 and 2008 in the Windsor area. Individuals in Windsor were asked which AQI (the U.S. or Canadian version) they tended to follow more closely. As can be seen in this table, individuals in Windsor are more likely to be familiar with the Canadian AQI (roughly 50% in both 2007 and 2008), and almost 30% follow only the American AQI. The data presented here is not meant to indicate a major issue with the public availability of the AQHI, but rather to highlight the fact that, even with a well-publicized AQHI in place, many Canadians living in border areas may consult U.S.-based air quality forecast information (which is based on different calculations and limited health-based information).
| And which of these two AQIs do you tend to follow more closely? | Windsor (2007) (n=64) % | Windsor (2008) (n=78) % |
|---|---|---|
| Windsor area – Environment Canada | 55 | 49 |
| Detroit area – Michigan | 30 | 26 |
| Both equally | 12 | 20 |
| Don't know/Not applicable | 3 | 5 |
Evaluation Issue: Performance
4a5. Participation in AQHI outreach to target populations
Indicator
Number/examples of participation in AQHI with target populations
Extent to which stakeholders indicate that the Program is engaged in outreach to target populations
Challenges/barriers to outreach
Rating
Some progress/ attention needed
There is evidence from regional progress reports and from interviews that AQHI is making strides in outreach to all target populations. There were comments across all regions, however, that pointed to the need for continued outreach to the health care community.
A review of regional progress reports31 indicated that the Program has made preliminary progress in terms of outreach to target populations.
All regions which provided progress reports to Program staff have implemented some form of outreach with health professionals and at-risk populations.
Four provinces (British Columbia, Nova Scotia, New Brunswick and Prince Edward Island) have engaged in outreach with educators and have taken steps to develop the capacity to answer inquiries from the public and media. British Columbia, for example, reported on a variety of outreach efforts with Health Care Partnerships (e.g., having AQHI "ambassadors" attend 2008 flu clinics in four interior communities to distribute brochures and interactive demonstrations of the http://www.airhealth.bc.ca/ website).
All but Quebec have engaged in outreach with the media
Manitoba and Saskatchewan have indicated to Program management that, at present, they have little regional capacity to conduct outreach or promotion activities beyond the presentation of AQHI data.
The views of stakeholders interviewed during the evaluation, however, were varied. Close to half of the individuals interviewed indicated that, since its implementation in 2007, the AQHI Program is making progress in terms of delivering outreach programs while the other half were unsure if progress has been made. A few interviewees believe that the AQHI is achieving this "to a limited extent," and a few believe that the AQHI is not reaching this objective at all.
Close to half of the interviewees specifically noted that this success is being achieved through direct outreach campaigns within the community. Examples, listed below by target population, were as follows.
General – All Target Populations
One interviewee noted that they provide community partners with materials such as posters and pamphlets to distribute at their community events.
One interviewee noted that they have developed a "tool kit" that includes PowerPoint presentations and newsletters for community agencies to use with client groups.
One interviewee noted that the AQHI has conducted outreach with health care providers, immigrant communities, child care services, and the elderly (i.e., sensitive populations).
Two interviewees noted that outreach is accomplished through attending conferences and other networking events, where they set up a booth to talk to people, distribute AQHI pamphlets and network with other organizations. Events like this were cited as providing an opportunity to share lessons, seek direction, and to receive input and advice on future opportunities from various stakeholders.
One interviewee also noted that train-the-trainer materials have been developed for use with client groups.
People with Existing Respiratory Conditions
Three interviewees noted that they work with a local Lung Association to hold community events that include AQHI.
Young Children
Other means of outreach mentioned include working directly with daycare centres, community groups, libraries, community organizations, and health units in order to target outreach to the population involved in activities such as summer camps and schools.
Health Community
A few interviewees specifically noted that outreach has included networking to build partnerships with the health community, particularly in the Maritimes and National Capital regions. This includes actively educating the public health community through building partnerships with health organizations that have their own network and programs through which they can do outreach.
Outreach that involved contact with health professional societies, hospitals and medical associations to provide AQHI information to doctors was also noted by two interviewees.
A minority of those interviewed noted that there is still work to be done in the area of outreach, particularly in the Ontario region. Those interviewees noted that broad uptake of the information probably has not been achieved through the outreach delivered to date; therefore, outreach efforts must be continued. They also noted that outreach to date in the province has been limited and that more work needs to be done. One interviewee noted that they tested the awareness of AQHI in the community after conducting an outreach session and found that those in the community could not recall what the AQHI was, therefore highlighting the need to continue with AQHI outreach. In addition, other interviewees in Ontario noted that their ability to conduct outreach was affected by budgetary restrictions. Other interviewees in other parts of the country noted that it might be too early to tell if outreach is having the desired impact since some outreach programs started in summer 2009.
A few interviewees noted that further engagement with the health sector is needed. Three of these interviewees noted that general physicians need to be further engaged to deliver AQHI information. Suggestions included:
AQHI information sheets provided for placement in these doctors' offices; and
Engagement of other health professionals, such as occupational therapists or physiotherapists, in order to ensure that air quality is considered in health care planning.
Interviewees noted that information for health care workers is in short supply. Health care workers are involved in counseling high risk individuals, but their tool-kit to help this group deal with air quality issues is small. When the projected forecast is for higher health risk levels, health care providers need to know how to help patients plan their level of activity and this advice needs to be built into patient care plans.
A minority of interviewees noted that partnerships with NGOs need to be further developed. In particular, these interviewees noted that while the Heart and Stroke Foundation is a partner, the level of engagement is not sufficient and further engagement needs to be encouraged. Others also mentioned the potential to develop partnerships with agencies that target specific high-risk groups such as those with asthma.
One interviewee also noted that partnerships with health organizations, such as the Canadian Medical Association, are helping to achieve this AQHI outcome through the implementation of specific education programs to cater to the demand for an e-learning course for health professionals. The development of a credentialed e-learning course for continuing education health science students at the University of British Columbia was also mentioned by Program staff as one early outreach success.
Program management indicated that outreach and partnership work with at-risk populations and their health care providers, as well as with hospitals, were all areas that required further work.
Many interviewees perceive that partnerships between AQHI and health providers (and organizations dealing with populations sensitive to air pollution) have enhanced participation in AQHI outreach. Many of these interviewees noted that partnerships, such as those with NGOs (e.g., lung associations and the Heart and Stroke Foundation), greatly assist in the ongoing communication and dissemination of AQHI information. Partners were also seen as being able to deliver outreach efforts to those in their specific target groups; hence able to collect feedback on the quality of AQHI information and lessons learned from particular target groups.
Evaluation Issue: Performance
4b. Progress towards meeting intermediate and final outcomes as identified in AQHI logic model
Indicator
Performance measurement data addressing intermediate outcomes
Extent to which stakeholders indicate that the Program is progressing towards meeting the Program's intermediate goals
Rating
~ Some progress/ attention needed
While the preliminary evidence suggests that the AQHI is generally on track to meet the Program's intermediate goals, detailed below, many interviewees commented that it was too early to accurately assess the Program's achievements at this level. Baseline data based on Program performance measurement provide an approximate overview of public awareness of AQHI-type information. However, these baseline data are based on the older AQI and are therefore less meaningful until additional data specific to the AQHI are collected with similar populations. At a minimum, the performance data highlighted here can be used as baseline data for any evaluative activity to support future decision making. Limited qualitative and quantitative data are available to assess the attainment of these outcomes, although at this stage the data are either preliminary or refer to baseline data collected as early as 2005 using the original AQI as the unit of analysis. Caution should therefore be exercised regarding drawing broad conclusions:
Increased Prevalence of Individuals Modifying their Behaviour in Response to Air Quality Issues
A minority of interviewees believe that the AQHI is progressing towards "increasing the prevalence of individuals modifying their behaviour in response to air quality issues". Half of those interviewed were unsure or not able to comment on the question related to modifying behaviour. Some were able to comment but noted that it is still too early to tell how well AQHI is progressing towards this outcome. Interviewees provided some anecdotal evidence regarding behavioural change, for example, one noted having received emails from people WHO have benefitted from the AQHI reporting and have changed their behaviour based on the AQHI rating.
Increased Awareness of the Availability of the AQHI
As indicated in previous sections of this report, the majority of external stakeholders interviewed indicated that, since the implementation of the AQHI in 2007, the Program has taken steps to increase awareness by both increasing the availability of AQHI products and services and by disseminating materials and/or advice on air quality and health. The two primary ways in which this increase has been realized has been through the Internet and the Weather Network.
Data from surveys that examined public awareness of the AQI indicated that roughly 20% of rural Canadians and 40% of urban Canadians in regions that receive the AQI recalled seeing or hearing AQHI information, though only 2% to 4% recalled today's air quality forecast.
Increased likelihood of seeking out AQHI Information
The federal AQHI website is operational and is available to all members of the public. Website hit analysis of the Toronto AQHI pilot project32 indicated that individuals were interested in seeking out AQHI information. For example, between April and November 2008, Toronto Public Health's website on air quality (which provides links to the AQHI) received 4366 visits.
Increased Knowledge of Appropriate Actions to Take in Response to Air Quality Issues
Just over half of interviewees believe that the AQHI is progressing towards meeting its longer term goal of increasing knowledge of appropriate actions to take in response to air quality conditions. A minority of these interviewees noted that guidance on actions to take in response to air quality conditions and attendant risk levels is embedded in the information disseminated on the AQHI, thus, with the dissemination of this material, the public must have some knowledge of appropriate actions to take.
A minority of interviewees believe that the AQHI has been limited in its progress in this area. One of these interviewees noted that the focus in AQHI messaging thus far has been on short-term protective measures, foregoing any consideration of long-term protective measures for those sensitive to air pollution.33 Another interviewee noted that progress in this area has been limited since confusion still exists between the AQI and the AQHI. This confusion interferes with people's ability to increase their specific knowledge about the AQHI.
Some interviewees commented that it is too early to determine whether the AQHI is having an impact in this area. One interviewee noted that this is due to the fact that there have not been many "major air quality events" that required the delivery of more urgent messaging (e.g., air quality so poor that children should not be playing outdoors).
Evaluation Issue: Performance
4c. Progress towards meeting Adaptation Theme intermediate and long-term outcomes
Indicator
Data from other comparable jurisdictions indicating increased use and/or awareness of risks based on AQHI-type information
Extent to which stakeholders indicate that the Program is progressing towards meeting intermediate Adaptation Theme goals
Rating
~ Some progress/ attention needed
As with program-specific intermediate outcomes, the analysis of the Adaptation Theme outcomes reported here are still speculative. A brief review of the academic literature noted that there is only limited evidence that AQHI-type information increases use of health-related products or awareness of health risks. That said, respondents did indicate that the AQHI can play a key role in linking air quality with health, particularly through continued and increasing exposure through various media sources.
Increased Use of Air Quality Information and Products
Many of those interviewed believe that the AQHI is working towards achieving its goal of "increased use of air quality information and products." They attributed the increased use of information and products to the engaging, visually appealing and user-friendly website that links health to air quality. Some responded that the Program is still in its infancy stages, and indicated the index is still not readily available to all regions due in large measure to political barriers and, to some degree, resource constraints such as the lack of monitoring stations.
The interviewees emphasized that Canadians are increasingly accessing the AQHI website and paying closer attention to air quality. Interviewees emphasized that this holds particularly true for at-risk groups that are utilizing the information in determining action with respect to outdoor activity.
A minority of interviewees, based largely in Ontario and Quebec, indicated that this outcome is only being achieved "to a limited extent" or is not being achieved at all. A lack of coverage in many municipalities (only five regions in Ontario and two in Quebec) due to continued insistence on using AQI and Info-Smog Quebec and lack of monitoring stations were reported. Interviewees also emphasized that more time is necessary in order to increase visibility of AQHI considering that it is still a fairly new program.
There has generally been contradictory evidence on the effectiveness of the use of large-scale public health education strategies and products, though observational studies in particular have pointed to positive health impacts based on the use of public health tools.34 A recent analysis of the use of the UV Index in Australia concluded that, while use has remained relatively low (~5%), there is a correlation between an understanding of the UV Index and its use.35 The authors suggested that even approaches to improve the display of the UV index may not increase its use if they are not matched by efforts to increase public understanding.
Increased Awareness of Risks Associated with the Impacts of Air Quality
The majority of interviewees felt that the AQHI is advancing towards the Adapation Theme goal of "increased awareness of risks associated with the impacts of air quality," though much of this evidence was anecdotal. Interviewees reported, for example, that the AQHI appears to engage the interest of the active and healthy population to help plan their outdoor activities. Some interviewees noted issues such as a lack of public interest because the level of air quality is generally favourable in most regions. As a result, there is less inclination to seek information on the risks associated with the impacts of air quality, considering that many do not view air quality as a problem.
Interviewees also reported that the AQHI has played an important role in increasing public awareness of how air quality can impact their life, in relation to the risks associated with bad air quality.
In Ontario and Quebec, the majority of interviewees believe that awareness of risks is only being achieved to a limited extent due to the lack of AQHI coverage in many areas of the province. A lack of an alert mechanism, confusion with multiple indices, and the relative newness of the Program were among other barriers discussed.
An evaluation of the National Skin Cancer Awareness Campaign in Australia did find positive impacts related to increasing awareness of risks associated with excessive exposure to the sun.36 This program was targeted specifically at Australian youth, and was resourced at roughly A$7million between 2005 and 2007. There are limited comparisons between this program and the AQHI, particularly given the targeted population and the fact that all resources were devoted to marketing in the Australian program. The evaluation did point out, however, that a targeted marketing effort could produce a modest increase in awareness of risks associated with an environmental condition. For example, the evaluation noted a rise in the number of Australian teenagers using sunscreen at the beach.
Evaluation Issue: Efficiency and Economy
5a. Are there more cost effective, economic and efficient means of achieving objectives under the AQHI
Indicator
Extent to which stakeholders indicate that the Program is cost effective, economic and efficient
Examples of areas of improvement
Program data demonstrating improved cost effectiveness and/or economy in achieving objectives
Rating
Some progress/ attention needed
The evidence suggests that the AQHI is cost effective and economic in achieving its objectives.
| Budget Item | 2007-2008 ($) | 2008-2009 ($) | ||||
|---|---|---|---|---|---|---|
| Budget | Expenditure | Deficit | Budget | Expenditure | Deficit | |
| Salary | 2,300,378 | 1,808,791 | 491,587 | 2,522,321 | 2,402,979 | 119,342 |
| Other O&M | 824,530 | 537,474 | 287,056 | 1,576,768 | 1,698,699 | (121,931) |
| Capital | 25,000 | 24,999 | 1 | 70,000 | 70,000 | 0 |
| G&C37 | 150,000 | 50,000 | 100,00038 | 440,000 | 431,000 | 9,000 |
| Accommodation | 282,265 | 282,265 | 0 | |||
| CSS | 392,571 | 392,571 | 0 | |||
| Total | 3,299,908 | 2,371,264 | 878,644 | 5,283,925 | 5,277,514 | 6,411 |
Table 4.3 provides a brief overview of the AQHI's budgeted versus actual expenditures. The data indicate that the Program had a budget of roughly $3.3 million and spent over $2.3 million during the first year of the Program's operation (2007-2008). The second year of implementation saw a near doubling of the size of the actual budget (to roughly $5.2 million) and a considerable increase in the use of grants and contributions ($440,000). The data also indicate that the Program ran a budget surplus in both years, however the budget surplus decreased in the second year of implementation. In 2007-2008, the Program spent roughly 75% of its budgeted resources, compared to nearly 100% of its budgeted resources in 2008-2009.
The budget items presented here are collapsed into broad categories to simplify data comparison from year to year and are based on the annual budget as provided by the Program.
Interviews with Program management and document review indicated that much of the budget surplus in the first year was due to the fact that the funding was delayed and required cash management. As a result, there were subsequent delays in funding recipients and delays in the hiring process.
The majority of interviewees commented that resources are being managed efficiently under the AQHI. The most prominent themes were that no other alternatives could achieve the same results for less cost; that management is always trying to do more with less; and that they have achieved results with the relatively modest amount of funding available.
Program management and staff indicated that using local partners to act as outreach agents on behalf of the Program has been cost effective by allowing the Program to access sensitive and targeted populations that they would not have otherwise been able to reach and to widen the network of government and NGO partners engaged in promoting and developing the AQHI. Program staff indicated that the approach is based, in part, on the success of the U.S. Environmental Protection Agency (U.S. EPA) AIRNow Program which divests much of its resources directly to state and local agencies. It should be noted, however, that quantitative comparisons are limited due to the fact that the AIRNow Program is supported by regulations
Some suggestions were made by external interviewees with respect to increasing efficiency. For example, interviewees recommended a greater push for media coverage in order to leverage the potential for greater "reach" through broadcast media via such channels as regular television reporting, radio stations and newspapers. Interviewees also reported that a greater effort is required to educate people on the difference between AQI and AQHI.
Evaluation Issue: Efficiency and Economy
5b. Has the AQHI been implemented as planned?
Indicator
Extent to which stakeholders indicate that the Program is being implemented as planned
Examples of barriers to implementation
Financial and other administrative data indicating that the Program is being implemented as planned
Rating
Some progress/ attention needed
The evidence indicates that the AQHI is on track to being implemented as planned, though with the important caveat that there are continued risks to implementation in Ontario, Quebec and, in particular, Alberta. Financial data from the Program indicate that, though the Program did not spend its full allotment in the first year of implementation, it presently appears to be on track to using all requested resources.
Several quantitative indicators used in other areas of this report suggest that the Program is on track to be implemented as planned. The Program is generally spending its allocated resources as of 2008-2009 (Table 4.3) and is on track to implementing the AQHI in more CMAs than originally targeted (Table 4.1).
The CESD found evidence of effective collaboration on the part of the Program, noting that, "from the beginning of the Air Quality Health Index development process, Environment Canada and Health Canada consulted with a wide variety of provincial governments and stakeholders, such as municipalities and non-governmental organizations, and have incorporated their ideas as appropriate."39
In response to whether the implementation is moving as planned on a national basis, the response was generally positive from the majority of interviewees. They indicated that the Program has been rolled out effectively in a short timeframe and that the AQHI is currently present in all but one province. Interviewees credited both EC and HC for doing an effective job in planning, bringing everyone in, and working "from the ground up."
Barriers to Implementation – Political
The majority of interviewees made reference to political barriers (whether in their own region or in others) as a roadblock to achieving full implementation on a national basis.
The insistence on maintaining current programs such as AQI and Info-Smog (Quebec) in the province of Alberta and in some parts of Ontario and Quebec was seen as a challenge by external stakeholders. In the Prairies, all interviewees reported that implementation is moving as planned; however, many also made reference to political barriers as reasons for preventing the full adoption of AQHI (two interviewees specifically mentioned Alberta). The majority of interviewees in Ontario also cited political barriers within the province, as some provincial jurisdictions continue to use AQI. In Quebec, all respondents made reference to the political barrier with the majority of provincial jurisdictions electing to use AQI and Info-Smog Quebec instead of adopting AQHI.
Barriers to Implementation – Monitoring
The majority of external and internal interviewees reported that implementation is moving ahead as planned. However, interviewees made reference to a lack of resources which constrained the maintenance of monitoring stations.
Program staff also noted that monitoring stations are an ongoing challenge. NAPS agreements with the provinces allow funds to be used for the purchase of monitoring stations, but not for ongoing maintenance or upgrades to the monitoring technology. The Program is reliant on provinces to maintain and update these stations and is investigating indirect approaches, such as the use of satellite imagery, to improve AQHI forecasting in those areas where provincial funds have not allowed for additions to the monitoring stations.
Barriers to Implementation – Engagement
Just over half of external interviewees noted that there are barriers to engaging partners. A few noted limits within the health sector due to competing priorities and complex health messaging for patients. Also of note were challenges associated with the time required to initiate changes to school curricula and to build trusting relationships with new partners.
A minority of interviewees noted that no additional partnerships need to be developed. These interviewees believe that appropriate partnerships have been developed in order to ensure that migration to AQHI has occurred, AQHI information is delivered to appropriate target groups, and outreach goals are achieved.
Evaluation Issue: Efficiency and Economy
5c. Has the Program successfully addressed the recommendations from the BAQS evaluation?
Indicator
Extent to which stakeholders indicate that the Program has successfully addressed recommendations from the BAQS evaluation
Administrative data indicating that the Program has addressed the recommendations from the BAQS evaluation
Rating
Achieved
The majority of interviewees reported that the AQHI successfully establishes a correlation between air quality and health and that the AQHI addresses the requirement for uniform air quality reporting through a nationally standardized program.
The CESD Audit indicated that the Program represented a consistent national approach to reporting air quality information.
The Program provided a detailed, and populated, performance measurement matrix to support this evaluation. This performance measurement data represented a notable commitment to Program reporting and transparency. Program management and staff indicated concern, however, that current constraints on conducting public opinion research limited the Program's ability to conduct a national assessment of AQHI awareness and use.
The following section provides overall conclusions regarding the relevance and performance of the AQHI.
The AQHI does appear to be a relevant federal program which addresses a legitimate need for consistent air quality information and is considered a relevant federal government program by key external stakeholders. One key caveat, at present, is that the Program is still transitioning from the existing AQI in key regions of the country, and the distinction between the two is not yet clear. Key highlights of findings leading to this conclusion include:
The review of scientific literature suggests that there is a growing consensus that air quality has a tangible impact on human health;
The AQHI is based on the consensus view that improved consistency in reporting and better linkages with health messages were required to improve the existing air quality forecasting system;
Stakeholders believe the AQHI is an important link between air quality and health, and establishes a nationally consistent approach to reporting on health risks associated with poor air quality; and
A variety of external stakeholders noted that there is concern that the AQHI is seen as a duplication of the AQI in key areas of the country. This issue appears to be due to a combination of the ongoing need to promote the AQHI and its potential improvements over the existing AQI to provincial/regional partners, and the Program's partial state of implementation at present. There was no indication however, with the notable exception of Alberta, that the AQI was considered the superior air quality information or forecast model.
The Program generally appears to be on track in meeting its stated goals and, based on the information available, appears to be cost effective and well managed. Key highlights of findings leading to this conclusion include:
At the request of provincial partners, the Program's approach was adapted to provincial implementation rather than by Census Metropolitan Area (CMA). Even with this shift in approach, if continued implementation occurs as planned, the Program will meet or exceed the Program's targeted roll out in over 27 CMAs. These CMAs, however, are not necessarily those that were originally planned (notable exceptions include any CMAs in Alberta). The provincially-focused implementation has also resulted in increased geographic availability of the AQHI (i.e., for locations such as smaller municipalities which have a lower population than CMAs).
Rural areas in Canada are beginning to be covered as a result of increased provincial focus, though there remain concerns about an absence of monitoring stations to support data collection in rural and remote areas. Baseline performance measurement data, while limited in terms of its use as a baseline measure for the AQHI, do point to greater levels of awareness and concern about air quality issues in urban areas compared to rural areas, suggesting that the Program may need to consider approaches to increasing AQHI outreach in rural communities.
The Program appears to be effectively engaged in the dissemination of AQHI products, as well as outreach to targeted populations. The MOA process, in particular, appears to have generated a considerable amount of locally driven products and outreach activities using the AQHI to promote health. There remains a general concern, however, that the Program has not yet engaged all health providers to the desired level, in particular physicians, hospitals and NGOs working with at-risk populations. The long-term impact of AQHI outreach activities on behaviour changes and actions in the general population and with at-risk groups is required to fully assess the impact of the AQHI, though this was not examined in depth in this evaluation given the early implementation of the AQHI.
It would appear too early to validly attribute intermediate outcomes to the Program's activities and outputs. That said, the Program has taken steps to produce early baseline measures of awareness and use. Additional, comparable, data will need to be collected to understand the full impact of the AQHI.
The Program appears to be cost effective, well managed and on track to meet its goals. That said, there is a risk of sporadic national implementation in light of comments and documentation noting delays in Ontario and Quebec, and non-participation of Alberta.
Other areas for improvement are: expanding the engagement of media, including local media sources, emerging media sources (e.g., social network sites, enhanced Internet delivery capabilities, push technologies) telephone access and radio and developing the Program's approach to communicating an advisory when there are higher levels of air pollution and higher risk levels.
The following recommendations are based on the findings and conclusions of the evaluation. Several issues and challenges which were noted during the course of the evaluation, such as the impact of changes in provincial leadership or the need for additional monitoring stations in key areas of the country, while important, are not under the control or jurisdiction of the Program. The following recommendations refer to actions that can be taken by the Program to address those areas that do fall within the Program's control. The evaluation recommendations are directed to the two responsible Assistant Deputy Ministers at Environment Canada (Meteorological Service Canada) and Health Canada (Healthy Environments and Consumer Safety Branch) in light of their responsibility for the overall management of the AQHI implementation. Where appropriate, the agency responsible for initiating the management action is identified to facilitate future follow-up.
One of the concerns expressed by both external stakeholders and Program management and staff was the potential duplication and confusion between the AQI and the AQHI in those areas where the AQI phase out was occurring slowly, particularly in Quebec and Ontario. The CESD Audit Report also noted this issue, stating that "working with the provinces on issues related to the total or partial phase-out of their existing air quality indices" was an ongoing challenge for the Program. The evidence collected in this evaluation was consistent with the CESD's conclusions. Furthermore, while the evidence does indicate that Program management have been open and inclusive as regards the participation of Alberta, that province has remained reluctant to participate in the Program. Again, findings from this evaluation were consistent with the CESD Audit's conclusions that a key challenge for the Program would be "managing the increased implementation risk, in terms of having a common air quality index in place across the county, given that one province has not participated in the…development."
(Joint EC and HC) Develop a strategy to ensure continued support to provincial partners, particularly Ontario and Quebec, to move towards full AQHI implementation. Alberta should continue to be engaged to the extent possible to ensure that all provinces are at least offered the opportunity to engage the Program.
The AQHI, as indicated in the Program's description, operates in a complex jurisdictional environment and must rely on the engagement of external stakeholders, particularly provincial governments given their jurisdiction over aspects of air quality monitoring and health care, to ensure the successful implementation of the Program. The evidence indicated that the Program has successfully developed partnerships, for example the development of the provincial MOAs, to promote the AQHI and conduct early outreach efforts with the public and within the health community. The majority of external stakeholders highlighted the partnership approach used by AQHI Program management as key to the implementation observed to date. The current partnership with the Weather Network was also provided as a success story by both external stakeholders and Program management. Nevertheless, many external stakeholders indicated that more engagement with a variety of media sources would be necessary to ensure that the Program continued to meet its objectives of raising public awareness and use of the AQHI, particularly in rural communities.
(EC) Develop a strategy to engage a variety of delivery channels, including local media sources, emerging media sources (e.g., social network sites, enhanced Internet delivery capabilities, push technologies), telephone access and radio. This strategy should include specific mechanisms to serverural communities.
While there is strong evidence that the redesigned health focus of the AQHI is based on sound epidemiological science, and that the roll out of health-related messaging, information and outreach is occurring as planned, there were consistent comments from both external stakeholders and Program management to improve the targeting of at-risk populations such as the elderly or individuals with chronic respiratory problems.
Develop a strategy to prioritize the engagement of at-risk populations, including the following steps: a) (HC) engage health care providers for at-risk individuals when developing partnership agreements (e.g., MOAs) with other government stakeholders and NGOs; b) (EC) in consultation with provincial partners and users, further develop the Program's approach to communicating an advisory when there are higher levels of air pollution and higher risk levels.
The Program has taken steps to ensure performance measurement activities are occurring, most importantly populating a detailed performance measurement matrix and ensuring that key evidence such as provincial progress reports were available for analysis and reporting to the greatest extent possible. However, there is a need to conduct additional performance measurement activities to support Program decision-making and to help promote the AQHI, particularly given the fact that the Program's current performance measurement system is largely based on the AQI.
(Joint EC and HC) Continue to refine the Program's performance measurement strategy, including: a) a revised performance measurement framework, b) analysis on the utility of the current baseline values and, c) update of the Program's current logic model
The AQHI represents a simultaneous investment in both meteorological and health science. While the evidence collected during the evaluation and during the CESD audit indicated that the Program has built a scientific foundation in both areas (including observation and forecasting improvements and the epidemiological science behind the AQHI), there is no indication that this work is completeand continuous improvement in both areas is still required.
a) (EC) develop a strategy to assess and improve the current AQHI observation and forecasting methodology, with input from external stakeholders, b) (HC) develop a strategy to consult with stakeholders, including academic researchers and health experts in areas related to air quality and health, on an approach(s) to assessing AQHI-related health science issues.
The Program's resources will sunset in March, 2011. A considerable amount of investment has already been made under the AQHI, such as expanding the availability of the AQHI across Canada, the development of partnerships with provincial agencies and the Weather Network and ongoing development to forecasting air quality information and assessing the health impacts of outdoor air quality. While the progress noted above should be considered positive progress towards building a sustained, national AQHI, it is not clear which, if any, Program activities could be sustained beyond the end of the current phase of federal funding. Furthermore, though issues such as the acquisition of additional monitoring stations or additional health research related to the health impacts of outdoor air quality are outside the purview of this current round of funding, these issues have tangible impacts on the development of the AQHI. It is also to be expected that achievement of public health objectives through promoting individual behaviour change requires a long-term, sustained strategy. The issue of sustainability was also noted by the CESD auditors, noting that while "the federal government allocated funding in 2007 to support the continued development and implementation of the AQHI, it will provide this funding only until 2011." The evaluation team recognizesthat the focus has been on initial program implementation and that there has been limited opportunity to examine the long-term issues. There is a risk, however, that without a clear approach to addressing the remaining issues, some of which are long-term, the Index's ongoing development will not be fully addressed."
(Joint EC and HC) The Program should develop a comprehensive sustainability plan to identify and begin to address long-term issues associated with maintaining the AQHI or develop an exit strategy to ensure work to date is sustained without federal support beyond 2011.
(Joint EC and HC) Develop a strategy to ensure continued support to provincial partners, particularly Ontario and Quebec, to move towards full AQHI implementation. Alberta should continue to be engaged to the extent possible to ensure that all provinces are at least offered the opportunity to engage in the Program.
In Quebec, an agreement amongst federal leads and Quebec partners will see the AQHI pilot expanded to Montreal, for the spring 2010. Federal support for the AQHI pilot assessment would continue until Spring 2011. The current Info-Smog Program would be recast for the spring 2011 featuring the AQHI for large urban communities and the AQI for those locations where there is inadequate monitoring. The hybrid Info-Smog Program would remain in play until such time as the Program is able to provide AQHI forecasts for those communities which are served by the AQI.
In January 2010, a modification of the Memorandum of Understanding with Ontario for the AQHI forecast partnership was negotiated which would free-up Ontario Ministry of Environment staff for an AQHI location/province-wide assessment. Financial support from federal leads is budgeted to support provincial decision-making regarding implementation.
Alberta remains the only province outside the AQHI implementation planning despite recent efforts to engage staff in AQHI public meetings. The Alberta government's recent decision to introduce a revised AQI (also scaled 1 to 10)for June 2010 complicates AQHI adoption in that province. A strategy for integrating the Alberta circumstance into the national rollout will be in place in March 2011.
Actions: Targeted approaches in each province (as detailed above)
Functional Responsibility: ADM Meteorological Service of Canada (MSC) and ADM Healthy Environments and Consumer Safety Branch
Contacts: Director, Water, Air and Climate Change Bureau, Health Canada and Director, MSC Operations – Atlantic
Timeline: Spring 2011
(EC) Develop a strategy to engage a variety of delivery channels, including local media sources, emerging media sources (e.g., social network sites, enhanced Internet delivery capabilities, push technologies) telephone access and radio. This strategy should include specific mechanisms to serve rural communities.
The prospect of Program sustainability is being enhanced through the ongoing work, to fold the AQHI into the dissemination pathways of the MSC. For example, a user specification document for the enhancement of AQHI on the Weatheroffice website was developed in the fall of 2009 and there have been negotiations for the implementation of enhancements over the next two years. The technology support for multiple voice products has been updated recently. This will support, for example, the provision of national AQHI forecast over the national telephone network by spring 2011.
In December 2009, Health Canada entered into a 3 year agreement with the Weather Network to fund AQHI promotion and information pieces through their various dissemination vehicles. These enhancements will be phased in over the period of the contract with a concerted push for increasing AQHI visibility in spring 2010.
Non-governmental partners have shown that there is a significant potential in using social media networks. In the April 2010, the best practices guide for AQHI outreach will be enhanced with a section on the use of social media networking tools as will the Program's outreach strategy which will be completed for summer 2010.
Actions: a) National AQHI forecast over the national telephone network; b) AQHI dissemination through Weather Network; c) section on social media networking tools in best practices guide and outreach strategy
Functional Responsibility: ADM Meteorological Service of Canada (MSC) and ADM Healthy Environments and Consumer Safety Branch
Contact: Director, MSC Operations- Atlantic
Timelines: a) spring 2011; b) spring 2010; c)summer 2010
Develop a strategy to prioritize the engagement of at-risk populations, including the following steps: a) (HC) engage health care providers for at-risk individuals when developing partnership agreements (e.g., MOAs) with other government stakeholders and NGOs; b) (EC) in consultation with provincial partners and users, further develop the Program's approach to communicating an advisory when there are higher levels of air pollution and higher risk levels.
A national approach is being formulated to promote the AQHI with health care professionals through national professional organizations, publications and conferences. Supporting these efforts will be a Health Canada funded University of British Columbia online course for medical professionals on the Health Effects of Air Pollution and the AQHI which has been available since September 2009 and will run for 2 more years. In the spring of 2010, the AQHI Program will embark on a review process of the AQHI health messages to begin to address issues which have been raised over message efficacy by our stakeholders. A workshop in spring 2010 will kick off a multi-year process dedicated to developing and communicating more effective health messages.
In May 2009,federal leads formed an AQHI Advisory working group to examine the issue of communicating to the general public, with a focus on at-risk individuals, when there are higher levels of air pollution and higher risk levels. This working group is made up of representatives from the provinces and is helping to plan an Advisory and Special Air Quality Statement pilot project in Nova Scotia for summer 2010 which will lead to the national implementation of an advisory program.
Actions: a) Online course on Health Effects of Air Pollution and the AQHI (University of British Columbia) and workshop dedicated to developing and communicating effective health messages; b) Advisory and Special Air Quality Statement pilot project in Nova Scotia
Functional Responsibility: ADM Meteorological Service of Canada (MSC) and ADM Healthy Environments and Consumer Safety Branch
Contacts: Contacts: Director, Water, Air and Climate Change Bureau, Health Canada and Director, MSC Operations – Atlantic
Timeline: a) Spring 2010; b) Summer 2010
(Joint EC and HC) Continue to refine the Program's performance measurement strategy, including: a) a revised performance measurement framework, b) analysis on the utility of the current baseline values and, c) update of the Program's current logic model.
Program staff have taken a leadership role under the Adaptation Theme with respect to logic model and indicator development. The development of a performance measurement and management framework is under way. The focus for the initial stages of the framework will be on measuring and managing performance with respect to the at-risk population, with a first draft available by June 2010.
Program principles showed significant foresight in conducting a national baseline survey in spring 2007 as well as numerous post-event surveys after smog advisories. These data are being used to establish baseline values for some of the Program performance indicators, however, more data needs to be collected, both to gather missing baseline data and to start ongoing performance measurement of indicators. Over the past few years, staff have been unable to collect this additional data because of the ongoing challenge of conducting public opinion research. This remains a clear impediment to measuring Program performance and outcomes. Qualitative measures and anecdotal evidence collected by partners provide important complementary information. These, however, in the absence of quantitative data, cannot provide a strong evidence base to support the continuous improvement objectives of the index. Also, ways to collect data that fall outside the definition of public opinion research are being pursued, and ways to fill in missing data as well as collect ongoing Program performance measurement data will continue. In addition, extensive re-analysis of existing data is currently being done to help establish the most accurate baseline values possible for performance indicators.
Program staff have been proactive with respect to development of a program logic model and indicators. The above-noted development of a program performance measurement and management framework will be supported by a revision in conjunction with the logic model. This will ensure that the program logic continues to accurately reflect the Program as implementation continues, based on the most recently available information, including the results of this evaluation. A first draft will be available for EC and HC management by June 2010.
Actions: a) Revised Performance Measurement Framework; b) re-analysis of existing data; c) Revised Program Logic Model
Functional Responsibility: ADM Meteorological Service of Canada (MSC) and ADM Healthy Environments and Consumer Safety Branch
Contacts: Contacts: Director, Water, Air and Climate Change Bureau, Health Canadaand Director, MSC Operations – Atlantic
Timelines: a) Summer 2010; b) ongoing; c) Summer 2010
a) (EC) develop a strategy to assess and improve the current AQHI observation and forecasting methodology, with input from external stakeholders, b) (HC) develop a strategy to consult with stakeholders, including academic researchers and health experts, in areas related to air quality and health, on an approach(s) to assessing AQHI-related health science issues.
A green paper will be prepared by the spring 2010 for consultation and will lay out the vision for an enhanced and sustainable AQHI and Forecast program. Informed by third-party evaluation, supporting documents, ongoing input from stakeholders and experiences of staff and management, the paper will identify a number of key areas where the existing Program can be enhanced. By virtue of their importance to the sustainability of the Program, improvements underpinning the forecast production scheme and the ongoing challenges presented by air quality monitoring will be part of this document.
Assessing the health science that contributes to the formulation of the AQHI is an important piece in keeping the AQHI relevant and up to date. As the scientific formula that underpins the AQHI used the most recent epidemiological data available at the time and completed peer review in 2004, there has not been enough new data available to warrant a full re-formulation of the AQHI. That said, Health Canada is committed to keeping the index up to date. Developing a strategy to consult with stakeholders, including academic researchers and health experts on how to best assess the impact of new health science, and how it might impact the formulation of the AQHI, is important. Health Canada will commit to developing such a strategy, with input from stakeholders, by March 2011.
Actions: a) Develop Green Paper to lay out vision and enhancement of Program areas such as AQHI forecasting process and air quality monitoring; b) develop a strategy to assess AQHI-related health science issues.
Functional Responsibility: ADM Meteorological Service of Canada (MSC) and ADM Healthy Environments and Consumer Safety Branch
Contacts: Contacts: Director, Water, Air and Climate Change Bureau, Health Canadaand Director, MSC Operations – Atlantic
Timelines: a) Spring 2010; b) Spring 2011.
(Joint EC and HC) The Program should develop a comprehensive sustainability plan to identify and address long-term issues associated with maintaining the AQHI or develop an exit strategy to ensure work to date is sustained without federal support beyond 2011.
Third party evaluations and audits have commonly pointed out that the Program's funding continues to be awarded on a sun-setting basis. The aforementioned green paper will provide the sustainable path forward but in the event that future funding does not go forward as planned or funds are significantly reduced beyond what is currently available, an appropriately measured exit strategy will be included.
Actions: Ensure that the Green Paper considers the cessation of the Program as an option.
Functional Responsibility: ADM Meteorological Service Canada (MSC) and ADM Healthy Environments and Consumer Safety Branch
Contacts: Contacts: Director, Water, Air and Climate Change Bureau, Health Canadaand Director, MSC Operations – Atlantic
Timeline: fall 2010
1 An eighth theme, Clean Community Partnerships, has not been implemented.
2 Status Report of the Commissioner of the Environment and Sustainable Development Air Quality Health Index Report (2009).
3 There are exceptions to this statement. The Ontario provincial government, for example, also monitors international air pollution.
4 http://www.WHO.int/mediacentre/factsheets/fs313/en/
5 For example, the AQI for fine particulates has a much lower threshold in Quebec than in Ontario, so for the same actual PM2.5 ambient concentration, the AQI could be poor in Quebec but only moderate in Ontario.
6 http://www.ec.gc.ca/cas-AQHI/default.asp?lang=En&n=22BA50A8-1
7 Status Report of the Commissioner of the Environment and Sustainable Development Air Quality Health Index Report (2009).
8 Health Canada/Environment Canada AQI Health Messaging Development Workshop, Ottawa, December 2-3, 2004. Facilitated by Yvon Gauvreau Group Process Consultants.
9 Air Quality Health Index 2008 Forecast Verification Overview – PowerPoint Slideshow.
10 List based on Program's Logic Model
11 A list of citations is provided in Annex D
12 Air Pollution and Health, The Lancet, Bruenkreef & Holgate, 2002, 360(19), p. 1233-1242
13 http://www.CMA.ca/index.cfm/ci_id/86830/la_id/1.htm
14 2007 Baseline Survey
15 35% should be treated as an upper bound estimate of public awareness as 22% of individuals not in a forecast region reported being aware of an air quality forecast. Further, the AQI gave advisory warnings which may have increased temporary public awareness. The AQHI does not provide advisory warnings..
16 http://www.aqve.com/documents/docCPEQ1.pdf
17 The spreadsheet detailing differences and similarities between MOAs is available in Annex D
18 Status Report of the Commissioner of the Environment and Sustainable Development (2009); Chapter 2; p. 48-49.
19 Ibid. p. 55
20 It should be noted that this statement does not reflect current New Brunswick implementation which includes Moncton and Fredericton – currently, only the northern portion of New Brunswick lacks monitors
21 MSC AQHI Functionality Changes on Weatheroffice: User Specifications for Weatheroffice. Document Version #1.2
22 CMAs are cities with populations greater than 100,000
23 Figure obtained via program documents and confirmed during a presentation by a Health Canada Representative to the Adaptation Theme Director General Management Committee in October, 2009.
24 This table is based on the 2009-2010 and 2010-2011 AQHI Work Plan provided by the Program.
25 "Pending" refers to those municipalities that have not yet agreed to implement the AQHI during this phase of funding but may do so in the future.
26 AQHI 2009-2010 Work Plan
27 At the time of the evaluation, the AQHI for Manitoba and Saskatchewan was available on the Weather Network
28 At the time of the evaluation, partners were still involved in developing the Quebec pilot.
29 Data based on a summary of provincial progress reports – the full summary can be found in Annex F
30 More information on "push" technologies can be found at: http://en.wikipedia.org/wiki/Push_technology
31 New Brunswick, PEI, Nova Scotia, Toronto Public Health, British Columbia and Quebec were available at the time of the evaluation
32 Final Evaluation Report: Air Quality Health Index GTA Pilot, 2008
33 It should be noted that the AQHI is intended only to be a short-term protection measure
34 Hornick, Robert. Making Sense of Contradictory Evidence Public Health Communication (2000) 1-19
35 Carter, Owen & Robert Donovan. Public (Mis)Understanding of the UV Index Journal of Health Communication (2007) 12:41-52
36 Evaluation of the National Skin Cancer Campaign Ipsos-Eureka: April 2008
37 The AQHI uses the Terms and Conditions of the Environment Canada class Grant and Contribution Program to fund contribution agreements due to the low level of materiality.
38 The $100,000 was unspent as a result of a departmental cap on contribution spending and was carried over to FY 2009-2010
39 Status Report of the Commissioner of the Environment and Sustainable Development (2009); Chapter 2; p. 48-49.