Skip booklet index and go to page content

Evaluation of the Security and Prosperity Partnership of North America (SPP) Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE)

Report Clearance Steps

Planning phase completed
February 2011

Lessons learned completed
April 2011

Report completed
April 2011

Report approved by Departmental Evaluation Committee
July 2011

Acronyms used in the Report

CACs
Criteria air contaminants
CAP
Clean Air Partnership
CARA
Clean Air Regulatory Agenda
CEPA
Canadian Environmental Protection Act
FCM
Federation of Canadian Municipalities
FCO
Fleet Challenge Ontario
FTE
Full-time equivalent
G&C
Grants and contributions
GHGs
Greenhouse gases
HDD
Heavy-duty diesel
MSC
My Sustainable Canada
NCDC
National Clean Diesel Campaign
NGO
Non-governmental organization
NOx
Nitrogen oxides
NRCan
Natural Resources Canada
O&M
Operations and maintenance
PAA
Program Activity Architecture
PM
Particulate matter
PPRE
Pilot Project on Reducing Emissions from Vehicles and Engines
SPP
Security and Prosperity Partnership of North America
U.S. EPA
United States Environmental Protection Agency

Acknowledgements

The Evaluation Consultant Team of Stratos, Inc. and Alison Kerry would like to thank Nicola Scahill and Manjit Kerr-Upal for their ongoing input, provision of documentation, and review of key deliverables during the evaluation process. They would also like to thank the interviewees who provided their insights on the pilot project.

Input from the federal government’s Evaluation Division was provided by Robert Tkaczyk under the direction of the Environment Canada Evaluation Director, William Blois.

The evaluation was conducted by Stratos, Inc. in association with Alison Kerry.

Return to Table of Contents

Executive Summary

Scope and Objectives of the Evaluation

A formal, independent evaluation is required to assess the relevance and performance of the Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE). The scope of the evaluation is to cover the three years of the program and all related project activities. This requirement fulfills the commitment made in the original program design for the PPRE. As per the requirements presented in the Directive on the Evaluation Function, this evaluation will address value for money by including clear and valid conclusions about the relevance and performance of the pilot program. As the funding is not ongoing, and there is no decision required on program renewal, recommendations are not part of this evaluation. Rather, the focus is on lessons learned that may inform related future work and regulatory development.

Pilot Project on Reducing Emissions from Vehicles and Engines

The PPRE was an Environment Canada initiative, under the Security and Prosperity Partnership of North America (SPP), funded at the level of $1.2 million over three years (2008-11). The objectives of this pilot project were for Environment Canada and the United States Environmental Protection Agency (U.S. EPA) to collaborate on initiatives aimed at reducing emissions from vehicles and engines in order to better utilize the capabilities of each agency, and to share information on innovative programs such as retrofitting diesel engines for lower emissions. Funding was required to work on collaborative initiatives with the U.S. EPA, and to work with stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.

Evaluation Methodology and Design

The evaluation addressed five key issues: (1) Continued need for the program; (2) Alignment with government priorities; (3) Alignment with federal roles and responsibilities; (4) Achievement of expected outcomes; and (5) Demonstrated efficiency and economy. To examine these issues, and the specific evaluation questions related to each issue, the evaluation employed three lines of inquiry: (1) document review; (2) literature review and comparison of similar programs; and (3) key informant interviews. The evaluation scope focused on Environment Canada’s pilot project to work with stakeholders (within Canada) to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.

Evaluation Conclusions

The key findings from the evaluation, detailed in the report, are summarized below.

Relevance

  • The pilot project and its respective activities are relevant and continue to be needed to address heavy-duty diesel (HDD) greenhouse gas emissions and criteria air contaminants produced by public fleets.
  • The pilot project is directly aligned with federal government priorities and the strategic outcomes of Environment Canada to take action on climate change and reduce air emissions.
  • The activities of the pilot program are aligned with federal roles and responsibilities.

Performance – Project Design and Delivery

  • Performance information is provided by funded partners, primarily at the output level, in a timely and relevant manner. Data pertaining to the project’s performance indicator are currently being collected and will be reviewed at the end of the pilot. The evaluation did not find evidence of a formal internal reporting mechanism to senior managers or federal partners, though information was shared throughout the project on an informal basis.
  • The project was designed appropriately and included a clear and plausible link between program activities, outputs and expected results. The resources allocated to this project were commensurate with the expected results.
  • The pilot project identified that access to funding and the ability to work with the variety of players who are involved with municipal fleets (e.g. ranging from municipal governments to subcontractors and third parties) may be barriers to implementing a similar project on a national scale.
  • On behalf of Environment Canada, project partners engaged municipal fleet operators regularly and communicated the outputs from project activities in a timely and relevant manner. Correspondence and communication within the federal community was informal and in some cases inconsistent.
  • The pilot project was successful in developing a best practices guide and a resources guide for municipal fleet managers, and in providing advice on how to move forward with addressing emissions from school buses.

Performance – Effectiveness

  • The pilot project successfully generated a number of important outputs, including a best practices guide and a resources guide, fact sheets, and multiple reports that increased the level of knowledge about and number of approaches to addressing the on-road HDD municipal fleet and school bus sectors. One planned output (a baseline and profile of municipal fleets in Canada) was not achieved. Data are not yet available to assess the achievement of the project’s expected outcome.
  • The evaluation did not find any negative unintended outcomes.
  • Environment Canada successfully partnered with organizations that were experts in the subject matter and/or maintained a comprehensive understanding of the key organizations and individuals within the municipal fleet sector. In one case, the Department successfully leveraged funding from an organization to support the project.
  • The evaluation found that the project is on track to achieving its intended outcome in an economical manner.

Lessons Learned

The evaluation identified a number of lessons learned, and suggestions on how to improve the delivery of similar projects complementary to regulatory development. These are detailed in Section 6, and are summarized below.

Key Lessons Learned from the PPRE:

  • Allocating an appropriate amount of time to designing a program/project can lead to relevant and valuable results. Environment Canada allocated one year to design a successful pilot project that was relevant and met a demonstrable need that was not duplicative of other federal efforts.
  • Partnering with organizations that maintain expertise and existing networks can lead to effective results. Environment Canada worked with non-governmental organizations (NGOs) that maintained existing networks and a comprehensive understanding of municipal fleets across Canada. This approach enhanced program efficiencies by creating immediate access to existing networks, and provided greater assurance that project deliverables were credible and were useful to the municipal fleet community.
  • When provided with the appropriate tools and funding, Canadian municipalities can engage in actions to reduce emissions from their fleet vehicles. The PPRE has shown that by building the capacity of municipalities, providing tools, and providing assistance in accessing funds, emission reductions can be supported and achieved.

Suggestions for Future Projects:

  • A one-window approach, providing information and resources to reduce emissions from all types of vehicles in the municipal fleet, would be beneficial. There is merit in addressing the emissions from all vehicle types that are present in the municipal fleet more broadly, rather than just focusing on HDD. Fleet managers deal with multiple types of vehicles and make integrated decisions that can best be informed and supported by a more comprehensive approach to addressing emissions from all vehicle types.
  • Long-term funding is required to ensure action is taken to reduce fleet emissions. The pilot project has increased the general level of awareness of municipal fleet managers and has provided a number of resources to reduce the emissions associated with municipal fleets. However, the pilot program has also underlined a need for long-term funding to implement solutions to reduce emissions produced by municipal fleets. This pilot project has demonstrated that long-term funding from sources with municipal jurisdiction should be identified.
  • The Government of Canada can play an important role in identifying innovative technologies for reducing vehicle emissions. The Government of Canada can play an important role in testing of technologies for reducing on-road vehicle emissions. The PPRE-funded project in Ontario led to the identification of retrofit technology that has a proven ability to reduce emissions produced by school buses.
  • Environment Canada is best placed as an enabler for reducing vehicle emissions. The evaluation highlighted that this pilot project was a unique opportunity for Environment Canada to lead a pilot project that identified a gap, was complementary to HDD regulations, and addressed greenhouse gases and air pollutants. However, other federal departments, such as Natural Resources Canada, are now better placed to lead comprehensive capacity-building programs in this area, because of their mandate and emerging work under the FleetSmart program.
  • A national emissions baseline for municipal fleets is required. The pilot project identified the need for a national emissions baseline for municipal fleets, but found it challenging to complete. In order to target cost-effective emission reduction activities and to measure change with municipal fleets, a baseline is required.
  • The impact of a capacity-building / outreach program is maximized when delivered jointly with a suite of like-minded programs, including regulations/standards. Although education and capacity building are important tools to help municipalities take action to reduce their emissions, a suite of complementary and reinforcing initiatives, including regulations, provides them with the push and pull required for taking substantive action within their jurisdictions.

Return to Table of Contents

1.0 Introduction

Introduction

The evaluation of the Security and Prosperity Partnership of North America (SPP) Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE) was undertaken from January to March 2011, in response to a commitment to conduct an evaluation of the original program design documentation submitted to Treasury Board Secretariat officials. The intent of this evaluation is to inform management in Environment Canada, as well as the Treasury Board Secretariat, about the success of the pilot project and lessons learned that may inform related future work.

For the PPRE evaluation, program staff in the Strategic Transportation and Policy and Analysis Section of Environment Canada’s Transportation Division, along with a representative from the Department’s Evaluation Division, led the evaluation and performed an oversight role that involved providing input and feedback to the consultant hired to conduct the independent evaluation; approved all deliverables outlined in the contract; and coordinated the management response to the evaluation report.

This report summarizes the evaluation process and findings, conclusions and lessons learned. The report is organized into five sections:

  • Section 2 describes the pilot project, including the governance structure, allocated resources and expected outcomes
  • Section 3 indicates the evaluation objectives, scope, issues and approach taken
  • Section 4 documents the evaluation findings
  • Section 5 presents the evaluation conclusions
  • Section 6 outlines the lessons learned

The report also includes a number of annexes that form the information base for the evaluation’s key findings and conclusions, including a list of reference material, literature review details, interviewees, and a summary findings table.

Return to Table of Contents

2.0 Background

The SPP is an Industry Canada-led initiative that provides a flexible means for dialogue, priority setting, collaboration and action on issues affecting the security, prosperity and quality of life of Canadians, Americans and Mexicans. It addresses diverse issues, such as border facilitation, the environment, food and product safety, and includes measures to improve overall North American competitiveness.

A total of $39.2 million was allocated to the SPP over three years (2008-09, 2009-10, 2010-11). Of this allocation, $6.2 million was provided to Environment Canada for three projects:

  1. Chemicals assessment ($4 million)
  2. A North American air emissions inventory ($1 million)
  3. The PPRE ($1.2 million)

An overview of the funding provided to Environment Canada can be viewed in Table 1 below.

Table 1: SPP Funding to Environment Canada
Initiative2008-092009-102010-11Total
Chemicals Assessment$1,190,000$1,390,000$1,420,000$4M
N.A. Emissions Inventory$163,000$418,000$419,000$1M
PPRE$120,000$540,000$540,000$1.2M
Total$1,473,000$2,348,000$2,379,000$6.2M

This evaluation relates solely to the PPRE.

Table 2 summarizes the resources allocated to the Transportation Division for PPRE activities.

Table 2: Resources allocated to the Transportation Division for PPRE activities
Funded ActivitiesFiscal YearTotal
2008-092009-102010-11
FTE0.511 
Salaries$50,549$107,764$107,764$266,077
Employee Benefits$10,109$21,553$21,553$53,215
O&M$52,770$73,009$73,009$198,788
G&C $323,664$323,664$647,328
PWGSC Accommodation$6,572$14,010$14,010$34,592
Total$120,000$540,000$540,000$1.2M

2.1 Profile

The PPRE complements the Clean Air Regulatory Agenda (CARA) by addressing emissions from existing vehicles and engines that could remain on Canada’s roads for up to 20 years.1 It targets on-road heavy-duty diesel (HDD) vehicles such as trucks and school buses, and explores how best to voluntarily reduce their emissions (greenhouse gases [GHGs] and criteria air contaminants [CACs]). Emissions from these pre-2007 diesel engines contribute significantly to local, regional and global air pollution. The Government of Canada carries out various emission reduction programs targeting HDD vehicles (mainly Class 8 freight vehicles), such as Transport Canada’s ecoFREIGHT program and Natural Resources Canada’s (NRCan’s) FleetSmart program. This initiative complements these programs by focusing on Canada’s existing on-road public HDD vehicles, such as municipal fleet vehicles.

2.1.1 Objectives and Expected Outcomes

The objective of the PPRE, as noted in its program design, is as follows: “Environment Canada and the United States Environmental Protection Agency (U.S. EPA) will collaborate on initiatives to reduce emissions from vehicles and engines to better utilize the capabilities of each agency, as well as share information on innovative programs such as retrofitting diesel engines for lower emissions. Funding is required to work on collaborative initiatives with the U.S. EPA, and to work with stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.”

The first year of the SPP pilot (2008-09) was dedicated to developing the program design, which included consultations with Transport Canada, NRCan and the U.S. EPA to gain knowledge and advice on how the pilot project should be designed. These consultations allowed Environment Canada to better understand what work was already being carried out in this area (e.g., ecoFREIGHT and FleetSmart) and gaps or opportunities that could be addressed through the pilot project. Based on this design phase, two initiatives emerged:

Reducing Emissions from Canada’s On-Road Heavy-Duty Diesel Municipal Fleet

The objective of this project was to increase knowledge, awareness and supporting tools for municipalities to reduce their emissions of GHGs and CACs, by highlighting emission reduction opportunities and identifying barriers and challenges to be overcome during the implementation of emission reduction strategies for fleet vehicles. To realize this objective, Environment Canada signed a Contribution Agreement with the Federation of Canadian Municipalities (FCM) in January of 2010, in order to enable the following activities:

  • Establish partnerships with municipal fleet managers’ networks, federal and provincial agencies, environmental and non-governmental organizations (NGOs) and private sector organizations.
  • Create a baseline of the municipal heavy-duty fleet within Canada, including a fleet profile, an overview of best practices, and identification of barriers and challenges for initiatives to green fleets.
  • Identify and create an online resource in both official languages for municipal fleet managers to access information on best practices available to achieve emission reductions.
  • Promote in both official languages the online resource and other products and services (e.g., publications) to reduce HDD emissions.
  • Deliver education and training workshops in collaboration with partners to increase the capacity of municipal fleet managers to make decisions about reducing HDD emissions within their fleet profile.
  • Develop a business case for emission reduction initiatives in heavy-duty fleets, to be included in a resources guide for fleet managers.

In addition to the work completed with the FCM, Bronson Consulting was contracted to conduct a scoping study of Canadian school bus fleet characteristics, in order to identify key considerations in the development of any clean air programs for school bus fleets.

Furthermore, Environment Canada’s Ontario Region provided funds to two not-for-profit groups to conduct similar activities:

  1. The Clean Air Partnership (CAP) – Funding was allocated to CAP to update information about the level of awareness of school bus emissions and actions that Ontario groups have taken to address these emissions. CAP also examined projects undertaken by the federal government and provincial governments, and policies adopted by school boards in other provinces, to reduce emissions and exposures associated with school buses.
  2. My Sustainable Canada (MSC) – Funding was allocated to MSC to implement two diesel-retrofit pilot programs in Ontario (for small municipal school buses and garbage trucks).

The expected outcome for this pilot project was2: Increased implementation by Canadian municipalities of actions to reduce emissions from municipal on-road HDD vehicles.

The main performance indicator for this outcome was: “Population weighted count of municipalities that participate or report through the FCM that they are implementing greening fleet initiatives consistent with the pilot project (i.e., pre-2007 HDD vehicles).”

Table 3 outlines the resources allocated to this initiative.

Table 3: Allocated Resources
Funded ActivitiesFiscal YearTotal
2008-092009-102010-11
* 0.1 FTE for the evaluation
** Includes $21,000 for the evaluation
Reducing Emissions from Canada’s On-Road HDD Municipal Fleet
FTEs0.5 FTE0.6 FTE*0.6 FTE* 
O&M$52,770$73,009$73,009**$198,788
G&C $148,664$148,664$297,328

Advancing Regulatory Action to Reduce Vehicle and Engine Emissions

Canada and the United States have agreed to work together under the Canada-U.S. Air Quality Agreement to reduce transportation-related emissions. Under the SPP, Environment Canada and the U.S. EPA proposed options to improve information sharing and build upon existing informal mechanisms through the Agreement.

The objective of this initiative was to reduce emissions from vehicles and engines, while enhancing pre-regulatory cooperation between Canada and the United States. Furthermore, this project aimed to demonstrate how pre-regulatory collaborative work between Canada and the United States could work, as well as whether there are lessons that could apply to future regulatory work completed by both countries. More specifically this project aimed to identify and complete testing and technical assessments for reducing emissions from vehicles and engines. Based on emerging priorities and the recognized importance of complementary policy and regulatory regimes for North America’s transportation sector, this project was designed to support the alignment of future regulatory measures for the on-road vehicle fleet.

The expected outcome of this project was intended to be:

  • Canada and United States collaboration on technical studies to support future regulatory actions for the on-road fleet.
    • Performance Indicator: Number of technical studies developed in collaboration with the U.S. EPA to support future regulatory actions for the on-road fleet.

Table 4 outlines the resources for this initiative.

Table 4: Allocated Resources
Funded ActivitiesFiscal YearTotal
2008-092009-102010-11
Advancing Regulatory Action to Reduce Vehicle and Engine Emissions
Salaries 0.4 FTE0.4 FTE 
O&M    
G&C $175,000$175,000$350,000

Given that the grants and contributions (G&C) funding mechanism could not be used and that regulatory alignment with the U.S. EPA on the heavy-duty fleet was quickly evolving into a major focus of work for Environment Canada (i.e., collaborative work and information sharing between the two governments was already taking place under the Canada–U.S. Air Quality Agreement), a decision was taken to not implement this project and to transfer some of the funds to the first initiative.

2.1.2 Stakeholders and Beneficiaries

The PPRE aims to reduce emissions from on-road HDD vehicles and engines within Canada. Its reach extends to a broad range of stakeholders and beneficiaries that include:

  • Canadian municipalities who own and operate fleets
  • Environment Canada staff and management involved with vehicle regulatory development and policy analysis
  • Canadian environmental NGOs
  • Other government departments (including Transport Canada and NRCan)

2.2 Governance

The PPRE was implemented and managed by Environment Canada’s Transportation Division, Energy and Transportation Directorate (ETD). The project was implemented by a policy analyst from the Strategic Transportation and Policy and Analysis Section hired as part of the PPRE, and management oversight was provided by the Manager, Strategies and Outreach, Strategic Transportation and Policy and Analysis Section.

No separate committee or management structure was established within Environment Canada to further oversee or coordinate the PPRE initiatives.

The overall accountability structure for this pilot is the same as for other Transportation Division programs, i.e., this project reports through the Director General, ETD, to the Environmental Protection Board. Results are included in the Program Activity Architecture (PAA) element:

  • Strategic Outcome 3, Clean Environment
  • Program Activity 3.2, Climate Change and Clean Air
  • Program Sub-Activity 3.2.1, Climate Change and Clean Air Regulatory Program
  • Program Sub-Sub-Activity 3.2.1.2, Transportation Sector Emissions

1 In response to the Government of Canada’s On-Road Vehicle and Engine Emission Regulations,which introduced a new regulatory framework under the Canadian Environmental Protection Act, 1999, emission-reduction technologies have been introduced on HDD engines (starting with the 2007 model year). New HDD engines will emit significantly fewer air pollutants, including particulate matter (PM) and nitrogen oxides (NOx). However, diesel vehicles built before 2007 will continue to emit comparatively high levels of NOx and PM over the next 20 years.

2 A Results Based-Management Framework was not developed for this program; therefore a logic model does not exist.

Return to Table of Contents

3.0 Evaluation Design

3.1 Purpose and scope

A formal, independent evaluation is required to assess the relevance and performance3 of the SPP PPRE. The scope of the evaluation covers the three years of the program from 2008 to 2011 and all related project activities. This requirement fulfills the commitment made in the program design for this pilot, which was submitted to and approved by Treasury Board officials in the summer of 2009.

As per the requirements presented in the Directive on the Evaluation Function, this evaluation addresses value for money by including conclusions about the relevance and performance of the pilot program. As the funding is not ongoing, and there is no decision required on program renewal, recommendations are not part of this evaluation. Rather, the focus is on lessons learned that may inform related future work and regulatory development.

3.2 Evaluation Issues and Questions

The evaluation provides conclusions on the following broad evaluation questions:

  • Relevance: Did the pilot project remain consistent with, and contribute to, federal government priorities and responsibilities and address actual needs?
  • Performance – Project Design and Delivery: Was the PPRE project designed and delivered appropriately?
  • Performance – Effectiveness: Has the project achieved its intended outcomes?
  • Performance – Efficiency and Economy: Were the most appropriate, economical and efficient mechanisms used to achieve outcomes?

As this pilot project concluded at the end of March 2011, and all results are therefore not yet available, the evaluation report focuses on outputs and early outcomes achieved, as well as lessons learned.

Table 5 presents the evaluation framework, which outlines the evaluation questions and indicators that were explored as part of this evaluation, as well as the methods utilized to collect evidence.

Table 5: Evaluation Framework

Evaluation Issue & Question

Indicators

Methods

Relevance

1. Was there a need for the project? Is there a need for the project to continue?

  • Demonstration of environmental need
  • Demonstration that the pilot complements, and does not duplicate, other similar programs (CARA, ecoFREIGHT, etc.)
  • Demonstration that the pilot addresses gaps in environmental need
  • Project activities and reach are connected to environmental need
  • Views of stakeholders and project managers on the connection of project objectives with environmental need
  • Document review
  • Literature review
  • Interviews

2. Is the project aligned with federal government priorities?

  • Project’s objectives correspond to recent/current federal government priorities
  • Project’s objectives are aligned with current departmental strategic outcomes
  • Evidence of alignment with SPP, CARA, ecoFREIGHT and FleetSmart
  • Document review
  • Interviews

3. Is the project consistent with federal roles and responsibilities?

  • Project mandate aligned with federal government jurisdiction
  • Views on the appropriateness of federal involvement
  • Document review
  • Interviews
  • Literature review

Performance – Project Design and Delivery

4. Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?

  • Existence of effective performance measurement plan
  • Extent to which performance data are reliable, timely and relevant
  • Extent to which performance data inform/support EC’s decision-making processes
  • Document review
  • Key informant interviews

5. Is the program design appropriate for achieving expected program results?

  • Plausible link between program activities, outputs, and intended outcomes
  • Clearly defined and understood governance structure, including program processes, roles, responsibilities and accountabilities
  • Program resources/capacity commensurate with expected program results
  • Views on the appropriateness of program activities, processes and governance structures
  • Document review
  • Key informant interviews

6. Are there any barriers or challenges that could affect the success of this project if it was to be implemented nationally or integrated into an existing national project?

  • Views of stakeholders and project managers
  • Evidence of documented barriers or challenges
  • Evidence of solutions or approaches to addressing barriers or challenges
  • Interviews
  • Document review

7. Have the results of the pilot project been communicated effectively? How could communications be improved?

  • Views of project managers, partners and recipients
  • Extent to which planned communications activities have been implemented as intended
  • Evidence of information sharing, distribution of project publications and reports, etc. to public fleet operations and other fleets
  • Interviews
  • Document review

8. What are the best practices and lessons learned as a result of the project?

  • Identified lessons learned and shared best practices among partners
  • Identified strengths and weaknesses
  • Views on strengths / best practices, weaknesses and lessons learned
  • Interviews
  • Document review

Performance – Effectiveness

9. To what extent have the intended outputs and outcomes been achieved as a result of the project?

  • Views of project managers, partners and recipients on achievement of outputs and outcomes
  • Analysis of available performance data on project’s expected results and performance indicator
  • Population-weighted count of municipalities, through FCM, that report  implementing green fleet initiatives
  • Document review
  • Interviews

10. Have there been any unintended outcomes, either positive or negative?

  • Presence/absence of unintended outcomes
  • Views on whether unintended outcomes occurred
  • Document review
  • Interviews

Performance – Efficiency and Economy

11. Has the project undertaken its activities and delivered products (e.g., technical studies) in the most efficient manner?

  • Analysis of project operational costs in relation to the production of outputs
  • Views on whether the cost of producing project outputs is as low as possible
  • Evidence of / views on whether there are alternative, more efficient ways of delivering project activities and outputs
  • Document review
  • Interviews
  • Literature review
  • Financial analysis

12. Has the project achieved, or is it on track to achieve, its intended outcomes in the most economical manner?

  • Views on whether good value is being obtained with respect to the use of public funds
  • Evidence of / views on whether there are alternative program models that would achieve the same expected outcomes at a lower cost
  • Document review
  • Interviews
  • Literature review

3.3 Evaluation Approach and Methodology

Three lines of inquiry were completed to collect evidence in order to address the evaluation questions and develop the evaluation findings:

  1. An in-depth review of all relevant documents provided by the organizations engaged in the PPRE was completed, and additional documents were requested where gaps were identified or where additional information was needed to supplement the existing evidence base. The project team reviewed each document to understand the mandated requirements, goals and objectives, governance structure, authorities, activities, outputs and outcomes for the pilot project and its components. The overall adequacy of the document evidence base (e.g., availability of performance information) was also assessed. All evidence from the documentation was examined against the defined evaluation issues. A bibliography of the documents reviewed is included in Annex 1.

    This data collection method addressed evaluation questions 1-12.

  2. A literature review was conducted to: (1) better understand the federal government’s approach to addressing HDD vehicle emissions compared to the federal approach in the United States; and (2) compare the pilot to other programs in Canada that aim to reduce HDD vehicle emissions through capacity building. Details on the literature reviewed are included in Annex 2.

    This data collection method addressed evaluation questions 1, 2, 3, 11 and 12.

  3. Key informant interviews with representatives from Environment Canada as well as key partners and stakeholders were either conducted in person (e.g., with Environment Canada staff) or by phone. Interviews provided information on the program components’ relevance and performance. A total of 8 interviews with 11 individuals were conducted. A list of the categories and the number of informants is included in Annex 3, as are the interview guides.

    This data collection method addressed evaluation questions 1-12.

Draft findings from the above lines of inquiry were integrated and presented to the managers overseeing the evaluation in Environment Canada, in order to validate key facts and address omissions and errors before completion of the final evaluation report.

3.4 Limitations

The PPRE was not complete at the time of the evaluation (the evaluation findings were arrived at in February 2011 and the pilot project ended in March 2011). As such, the evaluators have commented exclusively on the pilot’s ability to deliver its intended outputs and achieve its intended outcomes. Also, the PPRE will not receive any additional funding beyond this fiscal year. Therefore, the evaluation did not include a series of recommendations to improve the delivery and effectiveness of the PPRE. Instead, a summary of lessons learned has been included to inform the design and delivery of additional pilot projects that focus on addressing vehicle emissions.

The evaluation did not address the second initiative (i.e., Advancing Regulatory Action to Reduce Vehicle and Engine Emissions), as it was not implemented and no G&C funds were spent on this project.4 Therefore, the evaluation scope does not reflect the original program design as the second initiative was not examined.


3 As per the 2009 Treasury Board (TB) Policy on Evaluation, the evaluation should assess: Relevance: The extent to which the Program addresses a continued need, is aligned with Government priorities, and is aligned with federal roles and responsibilities; and Performance: The extent to which the Program has achieved its expected outcomes, and demonstrates efficiency and economy.

4 The Evaluation Team was informed that the second initiative was never launched formally and was not announced by the Minister of the Environment. Program staff indicated that G&C funding allocated to this initiative was not utilized for this project, although some of the funding was reallocated to the pilot project and the remainder was lapsed.

Return to Table of Contents

4.0 Findings by Evaluation Issue

This section outlines evaluation findings for each of the defined evaluation issues and questions. Findings and ratings are presented by evaluation issue for the PPRE, in keeping with the protocol of Environment Canada’s Evaluation Division, with specific comments included under each evaluation question.

4.1 Rating of Findings

Ratings have been provided to indicate the degree to which the PPREhas addressed the key evaluation criteria. The rating is assessed according to the following chart provided by Environment Canada’s Evaluation Division. A summary table of these ratings/findings is provided in Annex 4.

Table 6 presents the four ratings and their significance, which are used to assess each evaluation question and the level to which the program has met the goals or objectives related to each question.

Table 6: Evaluation Ratings Legend

Rating

Significance

Achieved

The intended outcomes or goals have been achieved or met

Progress Made; Attention Needed

Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed

Little Progress; Priority for Attention

Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis

N/A

A rating is not applicable

~

Outcomes achievement ratings are based solely on subjective evidence

4.2 Relevance – Continued Need for the Program

Evaluation Issues (and related questions)

Indicators

Overall Rating

1. Was there a need for the project? Is there a need for the project to continue?

  • Demonstration of environmental need
  • Demonstration that the pilot complements and does not duplicate other similar programs (CARA, ecoFREIGHT, etc.)
  • Demonstration that pilot addresses gaps in environmental need
  • Project activities and reach are connected to environmental need
  • Views of stakeholders and project managers on the connection of project objectives with environmental need

Achieved

Summary

The evaluation found that there was a clear need for the pilot and a continued need to reduce emissions from municipal fleets. These vehicles are a significant contributor to local air pollution and GHG emissions, and pose a health risk to vulnerable populations (e.g., school children). The evaluation found that there are regional programs that target this need, but they are not national in scope, do not focus exclusively on on-road HDD vehicles, and remedial actions are not widespread. There are also similar federal programs, but the pilot was designed to complement these initiatives and address a specific area (i.e., on-road HDD including GHG and CAC emissions). The PPREwas designed to target an identified gap in meeting the needs of municipal fleet operators nationally. Due to the noted need, the overall rating has been assessed as Achieved.

Municipalities operate HDD vehicles as a normal part of their operations, for services such as the transport of school children, garbage collection and snow removal. However, municipal fleets are a key contributor to the environmental and health impacts of municipal operations. Burning diesel fuel creates GHG emissions that contribute to climate change, and CACsthat contribute to smog and acid rain. On average, Canadian municipal fleets are responsible for three to five percent of a municipality’s total GHGemissions, and consume billions of litres of fuel each year. In addition, HDD vehicles such as school buses can expose on-board children to elevated levels of air pollutants (see sidebar), adding to their environmental exposure. In particular regions (e.g., British Columbia and southern Ontario), local air quality is of particular concern and international agreements have been established to address air quality (e.g., Canada–U.S. Air Quality Agreement).

Although federal and provincial programs address emissions from HDD vehicles (see Annex 2), the PPREwas designed to complement these programs and target on-road legacy public vehicles that will continue to emit GHGs and CACs for many years. The pilot was designed not just to promote technology and tools to reduce emissions, but also to address needs specific to municipalities with respect to obtaining support for action (e.g., demonstrated economic benefits, business cases for action, etc.).

As part of the PPREprogram design, Environment Canada consulted with federal partners and examined current initiatives to ensure the pilot project filled an identified gap. Partners indicated that “municipalities are committed to reducing emissions from their fleets, but are unfamiliar with the appropriate technological and behavioural adaptations.” Furthermore, the literature review completed for this evaluation noted that regional programs in British Columbia, Ontario and Nova Scotia work with industry and/or fleet managers to reduce emissions; however, these programs are not national in scope. Table 7 summarizes these regional capacity-building programs and compares them to the PPRE(see Annex 2 for details).

Table 7: Comparison of Capacity-Building Programs

Fleet Challenge Ontario (Municipal Fleet Review)

Funding source

Provincial government

Delivery partner

One NGO

Target audience

Municipalities

Types of vehicles

On-road vehicles
Off-road vehicles (vehicle selection)

Objective

To support green management decisions and demonstrate ways to improve operations, thereby reducing costs and GHG emissions

Mechanisms/activities

Fleet audits

Workshops for fleet managers

Networking/forums for fleet managers

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

--

Duration

2008 (pilot project)
2009 to 2012


BC Air Action Plan (Action #10 only)

Funding source

Provincial government

Delivery partner

One NGO

Target audience

Municipalities
Commercial sector

Types of vehicles

On-road vehicles

Objective

To help owners of commercial and public sector vehicle fleets improve their fuel efficiency and reduce emissions

Mechanisms/activities

Fleet audits

--

Workshops for fleet managers

--

Networking/forums for fleet managers

Toolkits and guides

--

Links to other resources on website

Testing of reduction technologies

Duration

2007 to 2011


Clean Nova Scotia - Fleetwiser Program

Funding source

Federal government

Delivery partner

One NGO

Target audience

Municipalities
Utilities (energy)

Types of vehicles

On-road vehicles

Objective

To create a cleaner, healthier environment by informing, enabling and inspiring Nova Scotian fleet operators to respect and consider the environment in all their choices

Mechanisms/activities

Fleet audits

--

Workshops for fleet managers

Networking/forums for fleet managers

--

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

Duration

2009 to 2011


PPREPilot Project

Funding source

Federal government

Delivery partner

Three NGOs
One consulting firm

Target audience

Municipalities
School bus operators and districts

Types of vehicles

On-road HDDvehicles
Heavy-duty diesel engines

Objective

To reduce emissions (GHGsand CACs) from Canada’s on-road HDDmunicipal fleet by increasing knowledge and awareness and supporting tools for municipalities

Mechanisms/activities

Fleet audits

--

Workshops for fleet managers

Networking/forums for fleet managers

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

--

Duration

2008 to 2011

Source

Evaluation Work Plan, February 2011
http://fmv.fcm.ca/Enviro-Fleet/

4.3 Relevance – Alignment with Federal Government Priorities

Evaluation Issues (and related questions)

Indicators

Overall Rating

2. Is the project aligned with federal government priorities?

  • Project’s objectives correspond to recent/current federal government priorities
  • Project’s objectives are aligned with current departmental strategic outcomes
  • Evidence of alignment with SPP, CARA, ecoFREIGHT and FleetSmart

Achieved

Summary

The PPRE’s objectives are clearly aligned with current federal government priorities, and with Environment Canada’s strategic outcomes to take action on climate change and reduce air emissions. The PPREwas found to be complementary to other federal programs such as CARA and ecoFREIGHT. However, it was found to be similar to NRCan’s FleetSmart program, although FleetSmart focuses broadly on GHG emissions and public and private vehicles, while the PPREfocused on both GHGs and CACs specifically for on-road public municipal fleets and school buses. Due to the alignment with federal priorities and outcomes, the overall rating has been assessed as Achieved.

When designed, the PPREwas aligned with the Government of Canada priority, identified in the 2008 Speech from the Throne, of “tackling climate change and preserving Canada’s environment by reducing greenhouse gas emissions 20 percent by 2020.”5 This commitment to addressing climate change is maintained in the 2010 Speech from the Throne.

The PPREis aligned with Environment Canada’s priorities and strategic outcomes, as evidenced in the following Reports on Plans and Priorities (RPPs):

  • At the program design stage, the PPREwas aligned with the 2008-2009 RPP priority “reducing GHG emissions and air pollution” and contributed to the Department’s strategic outcome, as noted in the 2008 PAA: Canadians and their environment are protected from the effects of pollution and waste (Transportation Sector Emissions); and
  • Currently, the PPREis aligned with the 2010-2011 PAA strategic outcome: Threats to Canadians and their environment from pollution are minimized (Climate Change and Clean Air Program Activity, Climate Change and Clean Air Regulatory Program Sub-Activity, Transportation Sector Emissions Sub-sub Activity – which has the expected result: Reduced air pollutant emissions from Canadian motor vehicles and fuels).

The PPREcomplements the Government of Canada’s CARA by addressing emissions from existing vehicles and engines that could remain on Canada’s roads for up to 20 years. CARA focuses on the On-Road Vehicle and Engine Emission Regulations to reduce emissions from new HDDengines (starting with the 2007 model year). However, diesel vehicles built before 2007 will continue to emit comparatively high levels of nitrogen oxides (NOx) and PM over the next 20 years. The PPREtargets the on-road legacy public HDD vehicle fleet, such as municipal fleet vehicles, and explores how best to voluntarily reduce their emissions. Reducing emissions from the in-use public HDD fleet addresses two important Government of Canada priorities: reducing emissions (GHGs and CACs) and improving the health of Canadians.

The PPREwas designed to complement other federal programs such as Transport Canada’s ecoFREIGHT and NRCan’s FleetSmart (details of these programs are provided in Annex 2). It clearly complements ecoFREIGHT, which works with the freight transportation industry to increase the uptake of technologies and practices that reduce fuel consumption and emissions of CACs and GHGs. However, there is some overlap with the FleetSmart program, which offers free practical advice on how energy-efficient vehicles and business practices can reduce commercial and municipal fleet operating costs, improve productivity and increase competitiveness. NRCan’s transportation programs, in keeping with their PAA, indicate that they “aim to reduce GHG emissions from on-road transportation by encouraging drivers and fleet managers to use energy-efficient purchasing, driving and vehicle maintenance behaviours.” This is similar to the Environment Canada strategic outcomes. However, the focus of the FleetSmart program is GHGs (not air pollutants as well), and more on training and retrofits to promote energy efficiency specifically related to GHG emissions, while the PPREfocused particularly on capacity building and tools specific to HDD public municipal fleets and addressed both GHG and CAC emissions. Also, the PPREfocuses solely on public fleets (municipal fleets and school buses), while the FleetSmart program focuses on public and private vehicles.

4.4 Relevance – Consistency with Federal Roles

Evaluation Issues (and related questions)

Indicators

Overall Rating

3. Is the project consistent with federal roles and responsibilities?

  • Project mandate aligned with federal government jurisdiction
  • Views on the appropriateness of federal involvement

Achieved

Summary

Managing emissions is a partnership between multiple government jurisdictions and stakeholders. There is a federal role and responsibility for the Government of Canada to regulate emissions from vehicles and engines, and thus engaging in pilots such as the PPREto complement such regulations is appropriate providing that federal authorities focus their efforts on facilitating and enabling such programs and projects with relevant partners. Responsibilities for building capacity with and reducing emissions from the in-use municipal fleet are also strongly aligned with provincial and municipal roles and responsibilities. Stakeholders viewed the federal role in this pilot as appropriate. Therefore, the overall rating has been assessed as ‘achieved’.

All levels of government have roles to play in addressing emissions from HDD vehicles. The environment is a matter of shared jurisdiction between the provincial and federal governments. Both levels of government derive their jurisdiction from distinct powers granted by the Constitution Act 1867. Both levels of government may have the authority to regulate the same emissions, albeit based upon different Constitutional authorities. Municipalities derive their power from the provinces.

The PPREbuilds on the federal role to address emissions nationally (through a group such as the FCM) and to target regions where cross-border air emissions are an issue (e.g., southern Ontario). However, the specific roles for working with municipal fleet operators and funding retrofits to reduce emissions from diesel buses and heavy-duty trucks is more strongly aligned with provincial and municipal roles. This is evidenced by the existence of similar provincially funded programs in British Columbia and Ontario (see Annex 2 for details). In fact, the literature review, in comparing the PPREto other relevant capacity-building programs in Canada, highlighted that these programs were mostly designed for a regional audience and implemented at the regional and local levels by NGOs.

Views from interviewees indicated that this kind of pilot was unique for Environment Canada to undertake, as complementary measures to regulations are of interest to the Department but not its primary focus. Stakeholders believed that the appropriate federal role is primarily to regulate air quality, and secondarily to provide national capacity building in order to support municipalities as well as to provide incentives for change (adoption of new technology).

4.5 Performance – Design and Delivery – Performance Information

Evaluation Issues (and related questions)

Indicators

Overall Rating

4. Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?

  • Existence of effective performance measurement plan
  • Extent to which performance data are reliable, timely and relevant
  • Extent to which performance data inform/support Environment Canada’s decision-making processes

Achieved

Summary

The evaluation found that a performance indicator was included in the original program design for the PPRE, but has yet to be reported on. Performance information has been provided by funded partners, primarily at the output level, in a timely and relevant manner. However, there is little evidence of a formal internal performance reporting mechanism to senior management and/or federal partners for the entire pilot project. As the pilot has ended, performance data are not required to support future decisions, but it is relevant to determine whether the pilot has achieved its expected outcome. As it is expected that Environment Canada / FCM will report on the stated performance indicator at the end of the pilot, and other performance data have already been reported, the overall rating has been assessed as Achieved.

A comprehensive performance measurement plan was not developed for the PPRE. Rather, the program design document presented the following anticipated results and performance indicator for the pilot project:

  • Anticipated Result: Increased implementation by Canadian municipalities of specific actions to reduce emissions from municipal on-road HDDvehicles.
    • Performance Indicator: Population-weighted count of municipalities that participate or report through FCM that they are implementing greening fleet initiatives consistent with Project 1.

Although this level of detail was considered appropriate by Treasury Board to report on this initiative, no performance information on the defined indicator has yet been compiled. However, the FCM is conducting a final survey of all participants within its Enviro-Fleets Pilot Project, which will produce the required performance information for this indicator.

The work conducted by the FCM under the PPREhad a number of defined performance measures to track outputs and outcomes. These were reported on regularly within its quarterly reports. In addition, the FCM is planning to conduct an end-of-pilot survey with municipalities to better assess the impact of the pilot project. This information was not available for the evaluation but is expected by the end of March 2011. The performance information provided by the FCM thus far was assessed to be reliable, timely and relevant to the project.

The other initiatives funded under the PPRE(i.e., CAP, MSC, Bronson Consulting study) did not have specific performance measures, but were required to submit deliverables as part of their contracts. Interviews with program staff confirmed that all deliverables were completed appropriately and in accordance with the terms and conditions presented in the contracts between Environment Canada and those who received PPREfunding.

Although performance information was collected and reported by the FCM, and contract deliverables were produced as planned, there is no evidence of formal internal reporting mechanisms to senior management or other federal partners on the status of the pilot project, including all of its activities (i.e., those completed by the FCM, CAP, MSC and Bronson Consulting). Program managers indicated that they shared the PPREoutputs and results with management and partners informally through meetings/briefings or by distributing the final deliverables to relevant colleagues.

4.6 Performance – Design and Delivery – Program Design

Evaluation Issues (and related questions)

Indicators

Overall Rating

5. Is the program design appropriate for achieving expected program results?

  • Plausible link between program activities, outputs and intended outcomes
  • Clearly defined and understood governance structure, including program processes, roles, responsibilities and accountabilities
  • Program resources/capacity commensurate with expected program results
  • Views on the appropriateness of program activities, processes and governance structures

Achieved

Summary

The evaluation found a plausible link between program activities, outputs and expected results for the pilot project. There was a clearly defined, understood and appropriate program design and governance structure for managing projects, including their deliverables, timing, funding, and reporting requirements. In addition, program resources and the capacity of internal staff and funded partners were commensurate with the expected results. Therefore, the overall rating has been assessed as Achieved.

The program design document is the foundation of the PPRE design and delivery. It clearly lays out the rationale; anticipated results and performance indicator; activities, deliverables, timelines and funding; and the management of the projects and requirements for evaluation. The first year of the pilot was spent consulting with partners and determining the most appropriate activities to be undertaken, not only to meet the goals but to fill necessary gaps. The program design document was produced after this first year. Interviews indicated that it was important to take this time to investigate what had been done, what was being done by others, and what was needed to ensure the PPREcould provide valuable outputs.

As a result, the PPREwas appropriately designed to use the available resources and capacity of funded partners, as expert delivery agents, to achieve the intended results. Although no logic model was developed, the link between the planned activities, outputs and intended outcomes was plausible.

Although there was little formal management or governance structure for the pilot, this is seen as appropriate given the small amount of funds, the oversight provided by the Strategic Transportation and Policy and Analysis Section, and the controls put in place to effectively manage the funded partners (e.g., via contracts or contribution agreements).

4.7 Performance – Design and Delivery – Barriers

Evaluation Issues (and related questions)

Indicators

Overall Rating

6. Are there any barriers or challenges that could affect the success of this project if it were implemented nationally or integrated into an existing national project?

  • Views of stakeholders and project managers
  • Evidence of documented barriers or challenges
  • Evidence of solutions or approaches to addressing barriers or challenges

Achieved

Summary

The evaluation found that the key barriers to implementing a national program to address HDD emissions resulting from municipal and school bus fleets are: a) funding; and b) ability to work with all components of a municipal / school bus fleet (sometimes composed of both public and private vehicles). There were specific challenges identified in component pieces of the pilot that could be addressed by a broader, national strategy, but some elements, such as determining the amount of GHGs produced by HDD vehicles within a municipal / school bus fleet, are difficult to determine due to the heterogeneous composition of municipal / school bus fleets. As the barriers were clearly identified, and some recommendations for how to address them were included in component reports, the overall rating has been assessed as Achieved.

Two reports produced from the PPREidentified specific challenges that should be considered in developing a broader program to reduce emissions from on-road municipal and school bus fleets. The Bronson Consulting report noted that any clean air program for school buses should be part of a comprehensive school bus health and safety program in order to achieve high success rates, and should be linked with a larger national strategy. Implementation challenges included operators’ knowledge gaps regarding emission issues, and funding constraints. The report noted that program delivery agents from various sectors identified that the optimal role for the Government of Canada in a national clean air for school buses program is that of facilitator rather than being the driving force.

The MSC report on school bus retrofit programs noted that the key barrier is funding, and that another barrier is the reluctance of some school bus companies to consider diesel retrofit technologies for their buses due to time constraints and, in some cases, confidence in specific technologies. Some operators believe that the availability of new technologies, and legislation requiring the transport sector to reduce diesel emissions from school buses over the next several years, will significantly reduce air pollutants and that other efforts are therefore not necessary.

All stakeholders interviewed believed that funding was the key constraint in rolling out a more comprehensive national program. In addition, the variety of players that make up municipal / school bus fleets (e.g., vehicles owned and operated by municipalities vs. vehicles subcontracted by municipalities), along with the variety of information sources and technologies that currently exist, were seen as barriers to action. Busy fleet managers need one integrated and easy-to-access source of information to facilitate adoption. Also, multiple jurisdictions need to be engaged to establish a longer-term and more comprehensive approach.

4.8 Performance – Design and Delivery – Communications

Evaluation Issues (and related questions)

Indicators

Overall Rating

7. Have the results of the pilot project been communicated effectively? How could communications be improved?

  • Views of project managers, partners and recipients
  • Extent to which planned communications activities have been implemented as intended
  • Evidence of information sharing, distribution of project publications and reports, etc. to public fleet operations and other fleets

Achieved

Summary

The evaluation found that there was extensive communication and information sharing with the target audience (municipal fleet operators) to promote and facilitate emission reduction actions, particularly as part of the FCM project, with FCMleading the majority of the communication and information-sharing activities. Therefore, the overall rating has been assessed as Achieved. It was noted, however, that information sharing within the federal community was informal and at times inconsistent.

Table 8 summarizes how the FCM and CAP have shared information with organizations.

Table 8: Evidence of Information Sharing by FCM and CAP as part of the PPRE

Delivery Organization
and Information Shared


Number

FCM

Number of surveys completed by municipal fleets

21

Number of inquiries received from municipalities seeking support on municipal fleets

50

Number of participants attending the webinars and workshops

207*

Number of municipal contacts in the network that was established

324

Number of contacts on the list generated by the FCM’s ClearVantage that were sent information on resources and educational opportunities**

1300

Number of resources guides distributed

60

Number of municipalities responding to impact survey planned for March 2011

Figures not yet available

CAP

Number of organizations to which the report was sent directly

10

Number of views on the CAP web page dedicated to the school bus project (e.g., views of fact sheets)

1146 page views***

* 140 municipal representatives and 67 non-municipal representatives. Does not include participants who attended the FCM’s EnviroFleets Workshop, who were not registered; estimated at 50 participants.
** Information was also sent to subscribers to the FCM weekly newsletter and Partners for Climate Protection members.
*** 818 views from unique IP addresses were reported.

In addition, Environment Canada indicated that the Department distributed project outputs (e.g., Bronson Consulting report, Clean Air Agenda’s Clean Air Partnership Report) to other federal partners (Transport Canada and NRCan) as a means of sharing information. However, some interviewees stated that they had not been informed of some deliverables, such as the program design document or FCM publications such as the best practices guide.

4.9 Performance – Design and Delivery – Best Practices

Evaluation Issues (and related questions)

Indicators

Overall Rating

8. What are the best practices and lessons learned as a result of the project?

  • Identified lessons learned and shared best practices among partners
  • Identified strengths and weaknesses
  • Views on strengths / best practices, weaknesses and lessons learned

Achieved

Summary

The evaluation found that best practices and lessons learned were provided as part of the component pieces of the pilot. A best practices guide was produced for municipal fleet managers, and advice was provided for addressing emissions from school buses. Therefore, the overall rating has been assessed as Achieved.

Elements of the pilot project identified best practices, lessons learned and/or challenges as a result of the pilot activities. The FCMproduced a best practices guide that aims to help municipal fleet managers find cost-effective ways to reduce emissions from heavy-duty vehicles. The guide focuses on best practices relative to overall fuel use (e.g., life cycle management, idle-reduction technologies, etc.). The FCM also produced a resources guide to help fleet managers access the resources they need to improve their fleet efficiency. The CAP, MSC and Bronson reports completed under the pilot provided advice and recommendations for moving forward in addressing CAC emissions from school buses.

In terms of moving forward from this pilot, a summary of the best practices and lessons learned is provided in Section 6 of this evaluation.

Interviewees highlighted that it is important to take time at the beginning of a new pilot project such as this to investigate what others are doing, and what is needed, to ensure the pilot/program meets a clear need and builds on existing programs and initiatives. Many indicated that pilot projects such as this are important for demonstrating what can be done with a finite amount of resources.

4.10 Performance – Effectiveness – Outputs and Outcomes

Evaluation Issues (and related questions)

Indicators

Overall Rating

9. To what extent have the intended outputs and outcomes been achieved as a result of the project?

  • Views of project managers, partners and recipients on achievement of outputs and outcomes
  • Analysis of available performance data on the project’s expected results and performance indicator
  • Population-weighted count of municipalities through FCM that report implementing green fleet initiatives

Progress Made; Attention Needed

Summary

The evaluation found that all but one of the planned outputs have been achieved under the pilot project (the FCM did not obtain sufficient responses to establish a baseline of municipal HDD fleets across Canada). There was a consensus from the evidence and interviews that the pilot project achieved its intent. As the pilot recently ended, it is too early to determine the extent to which this intended outcome has been achieved, although the evaluation did find some evidence to suggest that progress is being made. Information on the performance indicator “population-weighted count of municipalities through FCM that report implementing green fleet initiatives” is not yet available, although it is anticipated that this information will be received through a final assessment survey to be carried out by the FCM in March 2011. Therefore, the overall rating has been assessed as Progress Made; Attention Needed.

Table 9 summarizes the planned and delivered outputs under the PPREpilot project. It is evident from this table that nearly all of the planned outputs were delivered.

Table 9: Planned and Delivered Outputs for the PPRE

Planned Outputs

Delivered Outputs

FCM

Establish partnerships required to deliver an effective program that will meet the goal. Examples include municipal fleet managers’ networks, federal and provincial agencies, environmental NGOs and private sector organizations.

Create a baseline of municipal heavy-duty fleet within Canada including a fleet profile.

X

Overview of best practices and identification of barriers for greening fleet initiatives.

Identify and create an online resources guide for municipal fleet managers that will provide information on best practices available to achieve emissions reductions.

Promote the resources guide and other products and services.

Deliver two webinars and four education and training workshops in collaboration with partners, using resources identified to increase the capacity of municipal fleet managers to act.

Create a resources guide for reference and use by municipal fleet managers.

Develop a business case for emissions reductions initiatives in heavy-duty fleets, for inclusion in the resources guide.

CAP

Develop a brief report that:

  1. outlines steps taken to collect information on the demographics of school buses in Ontario, and summarizes the data collected;
  2. outlines the steps taken to determine familiarity with, and action on, the 2005 Ontario Public Health Association report School Buses, Air Pollution & Children’s Healthand/or its findings, among key staff within school boards, school bus operators, the public health sector, and health associations; and summarizes the findings;
  3. outlines the steps taken to research the language being used in school board policies and/or contracts related to school bus emissions in Ontario, British Columbia and Nova Scotia, and summarizes the findings;
  4. includes model language that can be incorporated into school board policies and/or contracts, related to school bus retrofits and other emission reduction options, for use in Ontario and across Canada.

Develop a four-page fact sheet on school bus retrofits, directed at school board staff and to be disseminated across Ontario.

Develop a four-page fact sheet on school bus retrofits, directed at school bus operators and to be disseminated across Ontario.

My Sustainable Canada

Investigate diesel retrofit technologies to determine which technology is most suitable for use in school buses, and test it in a pilot program for school buses in a community in northern or eastern Ontario.

Bronson Consulting

Develop a report that features a review of school bus characteristics and North American school-bus-focused clean air programs, provides an understanding of how clean air programs could affect the Canadian school bus sector, and summarizes the key considerations from these programs.

The anticipated result/outcome for the first initiative under the pilot project was: “Increased implementation by Canadian municipalities of specific actions to reduce emissions from municipal on-road heavy-duty diesel vehicles,” with the following performance indicator: “Population-weighted count of municipalities that participate or report through FCM that they are implementing greening fleet initiatives consistent with the pilot project (i.e. first initiative).” As the pilot has just ended, the degree to which the anticipated outcome has been achieved is not yet documented. The FCM will be conducting an end-of-pilot survey of municipalities that is expected to provide information to address this outcome. In addition, future FCM reports (expected circa 2012-2013) will provide information on how municipalities took actions with their on-road HDD vehicles (e.g., new applications to the FCM’s Green Municipal Fund and its Partners for Climate Protection annual measures report on actions taken related to the HDD fleet).

The following information indicates that progress has been made toward achieving the stated outcome:

  • The FCM indicated that it has increased the capacity of municipal fleet managers and other officials to adopt measures that reduce GHGs and CACs, reduce fuel consumption, and reduce costs in municipal fleets through training and knowledge resources. Evaluation of their webinars indicated that a vast majority of participants felt that the webinar met or exceeded their expectations and increased their knowledge in the topic area, and agreed that they would apply the discussed ideas, tools and strategies to their work. Evaluation of the FCM workshops indicated that all respondents agreed that the sessions were worthwhile.
  • The FCM reported that 39 fleet-related measures have been undertaken by municipalities, with an aggregated amount of $8 million invested in greening the fleet measures (this is part of the broader Partners for Climate Protection program and is not directly attributable to the PPRE). In addition, the FCM’s Green Municipal Fund has changed its criteria to allow for fuel switching and retrofit projects in municipal fleets; it is anticipated that projects of this type will increase based on the work initiated in the PPRE.
  • Information on best practices and resources has been widely distributed to the target audience (as noted under Section 4.8) to increase awareness and facilitate the implementation of actions to reduce emissions.
  • MSC implemented retrofits on 60 school buses and 6 garbage trucks in southwestern Ontario. This resulted in substantial emissions reductions from these vehicles (approximately a 40% reduction of PM, 75% reduction of hydrocarbons, and 60% reduction of carbon monoxide).

4.11 Performance – Effectiveness – Intended Outcomes

Evaluation Issues (and related questions)

Indicators

Overall Rating

10. Have there been any unintended outcomes, either positive or negative?

  • Presence/absence of unintended outcomes
  • Views on whether unintended outcomes occurred

N/A

Summary

The evaluation found no negative unintended outcomes.

There was no evidence of unintended outcomes identified in the documentary evidence. Interviewees noted the following two key unintended outcomes:

  • increased understanding of the challenges facing municipal fleet managers in implementing changes to address emissions; and
  • enhanced relationships and networks between non-traditional partners (e.g., EC, NRCan and the FCM).

4.12 Performance – Efficiency

Evaluation Issues (and related questions)

Indicators

Overall Rating

11. Has the project undertaken its activities and delivered products (e.g., technical studies) in the most efficient manner?

  • Analysis of project operational costs in relation to the production of outputs
  • Views on whether the cost of producing project outputs is as low as possible
  • Evidence of / views on whether there are alternative, more efficient ways of delivering project activities and outputs

Achieved

Summary

The evaluation found evidence that Environment Canada partnered with other organizations to achieve the program’s objectives in an efficient manner while maximizing limited resources to manage the program within the Department and leveraging funding from partner organizations (i.e., the FCM).

Therefore, the overall rating has been assessed as Achieved.

Table 10 presents the expenditures of the PPREagainst original allocations. These expenditure figures were provided by program staff.

Table 10 PPREAllocations and Expenditures

The financial data provided highlight that $116,664 in G&C funding available for the pilot project’s first initiative in 2009-10 was not spent. Program staff explained that this funding was not provided to Environment Canada until late in the fiscal year 2009-10 (released in September), and therefore the program partner (FCM) only had a limited part of the year to spend the contributions effectively. As a result, some of the allocated funds available to the pilot project were lapsed.

In terms of comparable federal government approaches to reducing diesel emissions, the literature highlighted that the United States maintains a more comprehensive approach to delivering its objectives concerning diesel emissions, through the U.S. EPA’s National Clean Diesel Campaign (NCDC), which provides access to approximately $200 million (between 2007 and 2011) in funding to manufacturers, fleet operators, air quality professionals, environmental and community organizations, and state and local officials, to reduce diesel emissions (see Annex 2 for details). In the absence of such a large funding envelope to address diesel emissions in Canada, the PPREutilized partnerships with not-for-profit and/or expert organizations to achieve the project objectives.

The evaluation highlights that the PPREwas implemented by funding not-for-profit and/or expert organizations to undertake specific studies and initiatives. The funded partners were chosen based on their expertise in the required area and their ability to deliver work effectively and efficiently. For example, the FCM was selected as an appropriate organization to work with based on their ability to access municipalities and fleet managers through the FCM’s existing networks. The funds provided to these groups were relatively small (ranging from $25,000–$170,000) and all the deliverables were produced on time and met the Department’s expectations. The project was managed efficiently by one FTE in Environment Canada and was designed to complement existing initiatives.

In addition, some leverage was achieved through the FCM initiative (e.g., $37,000 was provided to the project by the Green Municipal Fund, which resulted in a 14% increase in the original $230,000 in G&Cs allocated for the project). Leverage was also achieved by the FCM collaborating with Fleetwiser to co-promote projects, with FPInnovations, and with Fleet Challenge Ontario (FCO) to deliver the Ontario workshop (FCO contributed $1,000).

Interviewees noted that the funded projects and the work managed by Environment Canada were undertaken in an efficient and collaborative manner. For these reasons, the PPREwas assessed to be efficient.

4.13 Performance – Economy

Evaluation Issues (and related questions)

Indicators

Overall Rating

12. Has the project achieved, or is it on track to achieving, its intended outcomes in the most economical manner?

  • Views on whether good value is being obtained with respect to the use of public funds
  • Evidence of / views on whether there are alternative program models that would achieve the same expected outcomes at a lower cost

Achieved

Summary

The evaluation found that the project is on track to achieving its intended outcomes in an economical manner. All stakeholders viewed the PPREas a good use of public funds, and believed that the delivery approach was economical. Therefore, the overall rating has been assessed as Achieved.

As noted above, the delivery approach was seen as economical, as it used expert non-profit groups that completed the required tasks for a relatively low cost and in a timely manner. The FCM project was able to reach a wide audience across Canada, to ensure a strong and equitable reach.

Interviewees believed that the federal government, on its own, could not have completed the work in the time required and for the available budget. For instance, it would not have been able to reach the target municipalities economically without the FCM, which had a pre-existing national network and channel in place. The FCM also had established credibility with the target audience, facilitating uptake and engagement. In addition, the FCM built on existing regional initiatives in British Columbia, Quebec and Ontario, and leveraged its broader networks and resources through the Partners for Climate Protection program and Green Municipal Fund. One alternative suggested for enhancing the economy and reach of the FCM workshops was to implement a train-the-trainer approach rather than hold individual workshops.

As well, alternatives could have been used to conduct the studies completed by MSC, CAP and Bronson Consulting, but these would likely not have been as economical, as these groups had successfully and recently conducted previous related studies and were able to build on this work in an economical and timely fashion. One interviewee noted that an alternative such as funding municipalities to do outreach or retrofits may be problematic, as they may not have the same drivers for action (they may be more concerned about fuel savings than environmental benefits).

The literature review (see Annex 2) indicated that the PPRE’s delivery is consistent with the delivery of regional programs that are being implemented in Ontario, British Columbia and Nova Scotia. Key highlights include:

  • other capacity-building programs maintain a similar temporal scope of 2–4 years;
  • implementation by a non-governmental partner is the preferred delivery vehicle for capacity-building programs; and
  • the target audience for capacity-building programs continues to be municipalities, although in at least one case a program is working directly with the commercial sector.

The literature review also demonstrated that there are multiple options for developing capacity with the goal of reducing HDD emissions, including the following:

  • Conducting audits of municipal fleets, which focus on identifying retrofit opportunities by conducting an on-site audit of a municipal fleet.
  • Delivering technical workshops to fleet managers, which educate the managers on fuel efficiency or how to green their heavy-duty vehicle fleet.
  • Providing networking opportunities, which include enabling networking opportunities for fleet managers through online forums, workshops, informal/formal meetings, and so forth.
  • Developing guidelines and toolkits for fleet managers (on a range of topics), such as tools for identifying appropriate technologies, and information or data on best practices / approaches for reducing HDDemissions.
  • Product testing, which focuses on supporting demonstration projects for emission reduction technologies.

52008 Speech from the Throne

Return to Table of Contents

5.0 Conclusions

The evaluation found that the pilot project was relevant and addressed a clear need to reduce emissions from HDD municipal fleet vehicles. Furthermore, the PPRE’s objectives were found to be clearly aligned with current federal government priorities and Environment Canada’s strategic outcomes to take action on climate change and reduce air emissions, as well as complementary to other federal programs such as CARA and ecoFREIGHT. By maintaining an exclusive focus on reducing GHGs and CACs from on-road public HDD vehicles, the PRRE addressed a unique gap. However, it is similar to initiatives under NRCan’s FleetSmart program.

Recognizing that air quality management is a shared responsibility between multiple government jurisdictions and stakeholders engaging in complementary measures like the PPRE pilot was appropriate for the federal government. Beyond the pilot stage, responsibilities for building capacity and reducing emissions from the in-use municipal fleet are also strongly aligned with provincial and municipal roles and responsibilities.

The evaluation found a logical and appropriate program design for the pilot project, one that was built on an identified need and utilized organizations with relevant expertise to assist in implementation. There was evidence of a clearly defined and understood structure for managing projects, including a clear articulation of expectations for deliverables and reporting requirements to be prepared by organizations who received PPRE funding. The pilot was successful in achieving its intended outputs:  all but one of the planned outputs for the pilot project was completed. Regarding the planned output that was not completed, Environment Canada hoped that the FCM could help establish a national baseline of the municipal HDD fleet across Canada by designing and distributing a survey to municipal fleet managers, but the response to this survey was too low to be quantified.

The evaluation found evidence that Environment Canada partnered with existing organizations to achieve the program’s objectives in an efficient manner, and to leverage a small amount of funding (i.e., from the FCM) for project activities. Furthermore, all stakeholders viewed the PPRE as a good use of public funds, and that the delivery approach taken was economical.

The expenditure information provided for this evaluation indicates that the majority of PPRE funds were spent through internal departmental activities, or by distributing PPRE funds (through contracts or G&C agreements) to consultants and non-governmental organizations.

Despite developing a specific performance indicator for this project, the evaluation did not find evidence of formal reporting against this indicator, nor was a formal internal reporting mechanism to senior management or federal partners developed to communicate results of the project and report on its performance. Rather, performance information was provided by funded partners in a timely manner, and focused on outputs. As the pilot has come to a close, performance data are not required to support future decisions, but it is relevant to determine whether the pilot has achieved its expected outcome. Environment Canada is expecting that the results of an FCM follow-up survey, to be made available in the near future, will allow the department to report on the project’s performance indicator.

Return to Table of Contents

6.0 Lessons Learned

The evaluation identified a number of lessons learned, and suggestions to improve the delivery of similar projects that are complementary to regulatory development.

Key lessons learned from the PPRE:

  • Allocating an appropriate amount of time for designing a program/project can lead to relevant and valuable results. The evaluation noted that the first year of program delivery was dedicated to designing a program that was relevant and met a demonstrable need that was not duplicative of other federal efforts. In this case, Environment Canada consulted with relevant federal partners and commissioned a study to identify an appropriate gap to be addressed by the pilot project.
  • Partnering with organizations that maintain expertise and existing networks can lead to effective results. For this project, Environment Canada elected to work with NGOs that maintained existing networks and had a comprehensive understanding of municipal fleets across Canada. Smart decision making around who to partner with can not only enhance program efficiencies by creating immediate access to existing networks, it can also provide greater assurance that project deliverables will be useful and credible and that they can effectively achieve results.
  • When provided with the appropriate tools and funding, Canadian municipalities can engage in actions to reduce emissions from their fleet vehicles. According to the FCM, up to 45% of Canada’s GHG emissions are under the direct/indirect control of municipal governments, and municipal fleets are responsible for 3–5% of total municipal GHG emissions as well as significant local health impacts. The PPRE has shown that by building the capacity of municipalities, providing tools, and providing assistance with accessing funds, emissions reductions can be supported and achieved. One of the key drivers for municipalities is the cost efficiencies that can result from fuel savings, along with emissions reductions.

Suggestions for future projects:

  • A one-window approach, providing information and resources to reduce emissions from all types of vehicles in the municipal fleet, would be beneficial. There is merit in addressing the emissions from all vehicle types that are present in municipal fleets more broadly, rather than focusing on HDD. Fleet managers deal with multiple types of vehicles and make integrated decisions that can best be informed and supported by a more comprehensive approach. Developing a one-window approach / portal that consolidates multiple information sources and provides information that is applicable to emissions from all vehicle types would be beneficial to fleet managers.
  • Long-term funding is required to ensure action is taken to reduce fleet emissions. The pilot project has increased the level of awareness of a group of municipal fleet managers and provided them with resources for reducing the emissions from their municipal fleet. However, municipalities continue to be forced to do more with less and an enhanced understanding alone is not sufficient to ensure that action is taken to reduce emissions. This pilot project has demonstrated that long-term funding sources from other sources should be identified.
  • The Government of Canada can play an important role in identifying innovative technologies for reducing vehicle emissions. The Government of Canada can play an important role in testing technologies for reducing on-road vehicle emissions. The PPRE-funded project in Ontario identified retrofit technology that has a proven ability to reduce emissions produced by school buses.
  • Environment Canada is best placed as an enabler for reducing vehicle emissions. The evaluation highlighted that this was a unique opportunity for Environment Canada to lead a pilot project that addressed an identified gap, was complementary to HDD regulations, and addressed both GHGs and air pollutants. However, other federal departments such as NRCan are now better placed to lead comprehensive capacity-building programs in this area, because of their mandate and their emerging work under programs such as FleetSmart.
  • A national emissions baseline for municipal fleets is required. The pilot project was unable to develop a national emissions baseline for municipal fleets. In order to target cost-effective emission reduction activities and measure change with municipal fleets, a baseline is required. However, the municipal fleet operating environment includes multiple players, with some municipalities working with partners/third parties (e.g., private fleets) to provide services to their jurisdiction. Therefore, calculating an estimate of the emissions produced by municipal fleet vehicles across Canada requires an in-depth study of municipal fleet management and operating systems. Although emissions data are available for the 12 largest urban areas in Canada, little is known about the level of emissions in the remaining municipalities.
  • The impact of capacity-building / outreach programs is maximized when delivered jointly with a suite of like-minded programs, including regulations/standards. Although education and capacity building are important tools to help municipalities take action to reduce their emissions, a suite of complementary and reinforcing initiatives, including regulations, provides them with the push and pull required for taking substantive action within their jurisdictions.

Return to Table of Contents

Annex 1 Bibliography of Documents Reviewed

Bronson Consulting Group. Contract with Environment Canada, December 2009.

Bronson Consulting Group. School Bus Scoping Study: Fleet Statistics and Clean Air Programs, March 2010.

Clean Air Partnership. Contract with Environment Canada, June 2010.

Clean Air Partnership. Healthy School Buses for School Bus Operators, September 2010.

Clean Air Partnership. Healthy School Buses: School Board Factsheet, September 2010.

Clean Air Partnership. School Buses, Air Pollution and Children’s Health: Follow-up Report, October 2010.

Environment Canada. Clean Diesel Workshop: Solutions for Municipalities, March 2008.

Environment Canada. National Clean Diesel Campaign Workshop Agenda, June 2008.

Environment Canada. A Guide to Understanding the Canadian Environmental Protection Act, 1999, 2009.

Environment Canada. SPP Program Design, June 2009.

Environment Canada. Contribution Agreement between FCM and EC, September 2009.

Environment Canada. Contribution Agreement Amendment (between FCM and EC), May 2010.

Environment Canada. Reports on Plans and Priorities, 2008-2009, 2009-2010, 2010-2011.

Federation of Canadian Municipalities. Proposal to Environment Canada, August 2009.

Federation of Canadian Municipalities. Enviro-Fleet Progress Report 2009-2010, April 2010.

Federation of Canadian Municipalities. Green Your Municipal Fleet (Enviro-Fleet brochure), May 2010.

Federation of Canadian Municipalities. Enviro-Fleet Activities Report First Quarter, July 2010.

Federation of Canadian Municipalities. Enviro-Fleet Activities Report Second Quarter, October 2010.

Federation of Canadian Municipalities. Enviro-Fleet: Reducing municipal heavy-duty vehicle emissions – A guide to best practices, November 2010.

Federation of Canadian Municipalities. Enviro-Fleet Activities Report Third Quarter, January 2011.

Federation of Canadian Municipalities. Feedback forms for the Quebec, Ontario and BC Workshops, 2009-2010.

Fleet Challenge Ontario. Municipal Fleet Review Program website.

Fleetwiser. Fleetwiser website.

Government of British Columbia. BC Air Action Plan.

Government of British Columbia. BC Air Quality – Levels of Government Involved.

Government of Canada. Speech from the Throne, 2008.

Green Fleets BC. Green Fleets BC website.

My Sustainable Canada. Diesel School Bus Retrofit Pilot Project – Report, March 2010.

My Sustainable Canada. Proposal to Environment Canada, June 2010.

Natural Resources Canada. FleetSmart website.

Natural Resources Canada. Evaluation of the Transportation Sub-Sub Activity, 2010.

Transport Canada. ecoFREIGHT website.

Transport Canada. Sustainable Development Strategy, 2001-03.

United States Environmental Protection Agency. National Clean Diesel Campaign website.

Return to Table of Contents

Annex 2 Literature Review

A literature review was conducted by the Evaluation Team to help assess the broader relevance and success of the PPRE compared to other similar or related initiatives in the United States and Canada, and to identify alternative delivery mechanisms or best practices to be considered for similar programs that may be delivered in the future. The literature review was conducted alongside the document review and included a review of Internet-based information.

Two areas were explored:

  • Comparison of the Government of Canada’s approach to addressing emissions from HDD vehicles to the approach delivered by the U.S. federal administration:
    • A review of the U.S. NCDC and the Government of Canada’s ecoFREIGHT and FleetSmart programs was completed, to provide a more general understanding of the approaches taken by both countries to address vehicle emissions.
  • Comparison of the PPRE to other related capacity-building programs in Canada:
    • The Evaluation Team, in conjunction with the Project Authority, selected three capacity-building programs that aim to reduce vehicle emissions: (1) Fleet Challenge Ontario (Municipal Fleet Review); (2) BC Air Action Plan (Action #10); and (3) the Fleetwiser Program. The criteria used to select these programs included:
      • comparability of program’s scope and objectives to the PPRE objectives; and
      • comparable program structure and delivery mechanisms.

1. Comparison of the Government of Canada’s approach to addressing emissions from HDD vehicles to the approach delivered by the U.S. federal administration

The NCDC was initiated in 2000 by the U.S. EPA to promote clean air strategies, by working with manufacturers, fleet operators, air quality professionals, environmental and community organizations, and state and local officials to reduce diesel emissions. The NCDC focuses on reducing emissions from on-road and off-road diesel engines, through regulations, information-sharing mechanisms, regional clean diesel collaboratives, sector-specific programs and clean diesel funding. It is expected that the NCDC will award $200M USD in funding from 2007 to the end of September 2011. Among other programs, the NCDC administers a Clean School Bus program, which aims to reduce children's exposure to diesel exhaust and the amount of air pollution created by diesel school buses. Clean School Bus USA brings together partners from business, education, transportation, and public health organizations, focusing on anti-idling efforts, retrofitting old buses and replacing the oldest ones with new, less-polluting buses.

In Canada, ecoFREIGHT and FleetSmart are the two major programs with a mandate to reduce emissions generated by the transportation sector. ecoFREIGHT, established in 2007 and managed by Transport Canada, supports the freight transportation industry toward a greater uptake of technologies and practices that reduce fuel consumption, CACs and GHGs. The program was expected to receive $61 million from 2007 through to 2011. FleetSmart, a part of the ecoENERGY for Fleets suite administered by NRCan, seeks to provide practical advice and consultation on how energy-efficient vehicles and business practices can reduce fleet operating costs, improve productivity and increase competitiveness.

Table 11 provides a high-level overview of NCDC, ecoFREIGHT and FleetSmart. An overview of the PPRE has been included to highlight similarities and differences between the programs.

This analysis has highlighted that the United States has a comprehensive approach to reducing diesel emissions through the NCDC. The NCDC has significant funding ($200 million in clean diesel funding alone from 2007 to 2011) and is an integrated, umbrella system to channelling initiatives aimed at reducing diesel emissions. The target audience for this initiative is heterogeneous and includes manufacturers, fleet operators, air quality professionals, environmental and community organizations, and state and local officials, which allows the initiative to address a variety of systems and vehicle types, including on-road vehicles, existing and new off-road vehicles, marine vessels, and locomotives. Similarly, the NCDC includes multiple delivery mechanisms, including regulatory development, funding assistance, and development of publications and materials to develop the capacity of vehicle operators to reduce emissions caused by diesel vehicles.

The ecoFREIGHT and FleetSmart programs provide funding and capacity-building activities to reduce emissions, but they do not focus exclusively on diesel emissions. They each target a specific audience--the freight transportation industry for ecoFREIGHT, and the commercial highway freight sector for FleetSmart--and address off-road and on-road vehicles. Delivery mechanisms include capacity-building initiatives for both, but ecoFREIGHT also provides funding mechanisms and incentives in addition to establishing voluntary agreements with the freight transportation industry.

In addition to these programs, Canada has developed and will continue to develop a series of regulations to reduce smog-forming air pollutant emissions from vehicles and engines in alignment with the standards of the U.S. EPA. Currently, there are regulations in place to reduce air pollutant emissions from passenger cars and light-duty trucks, heavy-duty vehicles, motorcycles, small marine engines such as outboards and personal watercraft, recreational vehicles as well as construction and agricultural equipment, and garden equipment such as lawnmowers and chainsaws. Canada will continue to amend and update existing regulations to maintain alignment with the U.S.

Canada addresses diesel emissions under CEPA 1999 through:

  • The Sulphur in Diesel Fuel Regulations which came into force in 2003 set maximum limits for sulphur in on-road, off-road, rail (locomotive) and marine (vessel) diesel fuels.
  • The On-Road Vehicle and Engine Emission Regulations which came into effect in 2004, control emission of hydrocarbons, nitrogen oxides, carbon monoxide, benzene, and a number of other toxic substances from the operation of on-road vehicle engines. These regulations aligned Canadian emission standards with those of the United States for light-duty passenger vehicles, light-duty trucks (such as vans, pickup trucks and sport utility vehicles), heavy-duty vehicles (such as heavy trucks and buses), and motorcycles.
  • The Off-Road Compression-Ignition Engine Emissions Regulations which came into force in 2006 and introduced emission standards for diesel engines such as those found in construction, agricultural and forestry machinery. The standards were aligned with U.S. EPA tier 2 & 3 emission standards. On February 12, 2011, the proposed Regulations Amending the Off-Road Compression-Ignition Engine Emissions Regulations were published in the Canada Gazette Part I. These amendments will result in further reductions of smog-forming emissions from the off-road diesel engine sector.

The Government of Canada has also recently announced regulations to reduce greenhouse gas emissions from vehicles including:

  • The Passenger Automobile and Light Truck Greenhouse Gas Emission Regulations were published in Canada Gazette, Part II in October 2010 and establish progressively stringent fleet average GHG emissions standards for new vehicles sold in model years 2011 through 2016. These regulations are aligned with those in the U.S. and apply to companies that manufacture new cars and light trucks in Canada, or import these vehicles into Canada for the purpose of sale. In October 2010, the Government of Canada published a Notice of Intent in Canada Gazette, Part I, to develop standards for new passenger automobiles and light trucks of model years 2017 to 2025, in coordination with the U.S. EPA.
  • On May 21, 2010 Canada and the U.S. announced their respective intentions to develop regulations to limit GHG emissions from new on-road heavy-duty vehicles such as buses and combination tractors. The Canadian regulations would be developed under CEPA 1999 and would align Canadian standards with those of the U.S. for the 2014 and later model year vehicles.

Table 11: Comparison of U.S. and Canadian Approaches to Reducing Emissions from HDD Vehicles

U.S. Approach

National Clean Diesel Campaign

Overall objective

To promote clean air strategies by working with manufacturers, fleet operators, air quality professionals, environmental and community organizations, and state and local officials to reduce diesel emissions

Funding

N/A
($200 million for clean diesel funding from 2007 to 2011)

Lead organization

U.S. EPA

Delivery partner

U.S. EPA

Target audience

Manufacturers, fleet operators, air quality professionals, environmental and community organizations, and state and local officials

Types of vehicles

On-road vehicles
Off-road vehicles
Large ocean vessels
Diesel engines (including locomotive and marine engines)

Activities

  • Regulations
  • Information-sharing through website (e.g., tool kits, information on technologies, etc.)
  • Regional clean diesel collaboratives (through public-private partnerships)
  • Sector-specific programs (clean school buses, clean ports, clean construction, clean agriculture, SmartWay Transport)
  • Clean diesel funding, including:
    • National Clean Diesel Funding Assistance Program (50% of funding dedicated to public fleets)
    • National Clean Diesel Emerging Technologies Program
    • SmartWay Clean Diesel Finance Program
    • State Clean Diesel Grant Program

Duration

Ongoing since 2000


Canadian Approach

FleetSmart (part of ecoENERGY for Fleets)

Overall objective

To offer free practical advice on how energy-efficient vehicles and business practices can reduce fleet operating costs, improve productivity and increase competitiveness

Funding

N/A

Lead organization

NRCan

Delivery partner

NRCan

Target audience

Commercial highway freight sector (commercial and municipal fleets)

Types of vehicles

On-road vehicles

Activities

  • Capacity-building initiatives
    • SmartDriver training
    • Fuel Management 101 workshops
  • Information-sharing through website (fuel-saving information, tool kit, publications, links, newsletter)

Duration

Ongoing since 1997


ecoFREIGHT

Overall objective

To support the freight transportation industry toward a greater uptake of technologies and practices that reduce fuel consumption, CACs and GHG emissions

Funding

$61 million

Lead organization

Transport Canada

Delivery partner

Transport Canada

Target audience

Freight transportation industry (trucking industry, companies in the air, rail, road and marine sectors, users of the freight system)

Types of vehicles

Off-road vehicles
On-road vehicles

Activities

  • Six initiatives, including:
    • Freight Technology Demonstration Fund
    • Freight Technology Incentives Program
    • Marine Shore Power Program
    • National Harmonization Initiative for the Trucking Industry
    • ecoFREIGHT Partnerships
    • ecoENERGY for Fleets (NRCan program)
  • Voluntary agreements with freight industry associations
  • Information-sharing through website (case studies, ecoFREIGHT announcements and newsletters)

Duration

2007 to 2011


PPRE Pilot Project

Overall objective

To reduce emissions from Canada’s on-road HDD municipal fleet by increasing knowledge, awareness and supporting tools for municipalities

Funding

$1.2 million for 2008 to 2011

Lead organization

Environment Canada

Delivery partner

FCM
MSC
CAP

Target audience

Municipalities
School bus operators and districts

Types of vehicles

On-road vehicles (heavy-duty only)
Diesel engines

Activities

  • Capacity-building initiatives:
    • Enviro-Fleets project with the FCM (resources guide, best practices guide, webinars and workshops for municipal fleet managers)
    • Southwest Ontario school bus and garbage truck retrofit pilot
  • Studies and information-collection activities:
    • Scoping study on the HDD school bus fleets within Canada
    • Study on the number and age of school buses operated by or for school boards in Ontario
    • Research on policies and/or contract language that can be used to reduce emissions of GHGs or air pollutants from school buses

Duration

2008 to 2011

Source

Evaluation Work Plan
Documentation Review

2. Comparison of the PPRE to other related capacity-building programs in Canada

Fleet Challenge: Delivered by Fleet Challenge Ontario (FCO), an Ontario-based NGO, the municipal fleet review program is funded by the Government of Ontario. The objective of the program is to reduce GHG emissions from municipal fleets by supporting green management decisions and demonstrating ways to improve operations. The program has helped renew fleets in approximately 40 municipalities, by supporting initiatives such as developing tool kits and guidelines for municipal fleet operators, completing retrofit audits of on-site municipal fleets, and providing networking opportunities to municipal fleet operators by delivering workshops and hosting meetings. The program was launched as a pilot program in 2008 and is set to continue until 2012.

B.C. Action Plan: This is part of the B.C. provincial government’s broader strategy to lead the world in sustainable environmental management. The BC Air Action Plan contains eight actions aimed at making heavy-duty vehicles cleaner. For example, Action #10, “Green vehicle fleets: we’re making it happen,” pertains to Green Fleets BC, a partnership initiative led by the Fraser Basin Council. The goal of the initiative is to help owners of commercial and public sector vehicle fleets improve their fuel efficiency and reduce emissions. Under this initiative, the Council has been a key information hub for the latest on green technologies for private and public sector fleets, including taxis, emergency vehicles, delivery vans and commercial freight trucks. Green Fleets BC has also supported demonstration projects on trucking technologies and on the use of biodiesel in British Columbia’s public sector and commercial vehicle fleets, and sponsored a new network of fleet managers.

Fleetwiser: The Fleetwiser program in Nova Scotia is funded by NRCan and delivered by Clean Nova Scotia, a regional NGO. The program is currently helping four municipalities and one utility reduce their emissions and make environmentally friendly choices, by providing participating fleet managers and operators with free consultation, training and resources. These resources and training have included Fleet Efficiency Tool Kits developed for fleet managers, operators and technicians, and Fuel Management 101 workshops.

A summary of each program is provided in Table 12. In addition to these programs, a brief overview of the PPRE has been included to highlight the similarities and differences across these programs.

Table 12: Comparison of the PPRE to other capacity-building programs in Canada

Fleet Challenge Ontario (Municipal Fleet Review)

Funding (source and amount)

Supported by the Government of Ontario (Ontario Ministry of Transportation and Ontario Ministry of Finance)

$ - N/A

Partners

FCO (Canadian Energy Efficiency Alliance)

The FCO program has been made possible through the Canadian Energy Efficiency Alliance in partnership with the Ontario Ministry of Transportation and the Ontario Ministry of Finance, through the Strengthening Our Partnerships initiative.

Target audience

Ontario municipalities: 11 municipalities selected to participate in 2009; 8 municipalities in 2010; 10 municipalities in 2011

Between 2008 and 2010, over 7000 vehicles in Ontario’s municipal fleets have completed FCO’s Fleet Review.

FCO also works with governments, private business, and private and public sector fleets.

Context (rationale and need for the program)

It is estimated that approximately 26% of Canadian GHG emissions are released from the transportation sector (GHGs are directly related to fuel consumed). Automotive technology is improving in order to reduce emission intensity, but the number of vehicles on Canadian roads continues to grow.

Municipal fleet operations usually perform according to fixed annual cycles. Vehicles are outfitted to complete specific tasks and sustain municipal services. The Municipal Fleet Review Program delivers a comprehensive set of reports that support green management decisions and demonstrate ways to improve operations, thereby reducing costs and GHG emissions.

Objective/goal

The Fleet Review is a green fleet optimization program designed with principles similar to LEED© for buildings, and is Canada’s leading initiative in this area. Supported by the Government of Ontario, FCO works with each municipality’s fleet management team to evaluate a broad range of management options, including fuel efficiency, vehicle selection, maintenance techniques, asset management, computer systems and end-of-cycle disposal of surplus vehicles.

Mechanisms/activities completed

Fleet Review Process
  • The Fleet Review Program is applicable to licensed, on-road leased or owned vehicles using any fuel type.
  • Participating municipalities are required to contribute a nominal fee to the Municipal Fleet Review Program based on the fleet size (from $1,250 to $2,500).
  • Steps for the fleet review:
    • A compilation of fleet data by the fleet department. FCO employs a unique software system with an extensive database of current, relevant fleet information. A fleet’s data are entered and a series of computer-generated reports is produced. Results are further analyzed and interpreted by FCO experts in accordance with a fleet’s unique features.
    • Participants receive a set of detailed reports and a synopsis; contents include recommendations and a comprehensive roadmap to green and improve fleet efficiency. FCO experts then meet with the fleet operators and related personnel at their location to review and discuss the results and answer questions.
Forums and workshops

As part of the Ontario Municipal Fleet Review Program, and with the support of the Government of Ontario, FCO presents forums and workshops that keep fleet managers on the cutting edge of green transportation trends, technology developments and green fleet initiatives. These have included FCO Networking Breakfasts (two each in 2009 and 2010), an FCO-FCM workshop for fleet managers, etc.

Time frame / duration

Pilot project in 2008
Extension from 2009–2012


BC Air Action Plan

Funding (source and amount)

Funded by the BC Ministry of Environment ($28.5 million over three years to implement the plan, in partnership with industry, communities and other levels of government)

Partners

Action #10: Green Fleets BC, a partnership initiative led by the Fraser Basin Council

Target audience

For Actions 5–13: a variety of stakeholders (municipalities, school districts, transit organizations, etc.)
Action #10: Private and public sector fleets

Context (rationale and need for the program)

The BC Air Action Plan is part of the BC provincial government’s broader strategy to lead the world in sustainable environmental management, with the best air and water quality and the best fisheries management. The 28 actions outlined complement the work under way across B.C. to address climate change, improve air quality, protect water, enhance public transit, improve local planning, and make the province electricity self-sufficient by 2016.

Rationale for action on heavy-duty vehicles: older commercial and industrial vehicles can emit up to 60 times more fine PM than those with new, modern engines. And modern engines can run even cleaner when they switch from 100% diesel fuel to a biodiesel blend.

Objective/goal

The BC Air Action Plan contains eight actions aimed at making heavy-duty vehicles cleaner (Actions 5–13).

Mechanisms/activities completed

Actions:

Action #5: Retrofit HDD vehicles
Action #6: Retrofit transit buses
Action #7: Clean up school buses (funding to school districts to buy new, clean buses)
Action #8: Get AirCare ON-ROAD out to more communities
Action #9: Get big diesels to stop idling
Action #10: Green vehicle fleets: we’re making it happen
Action #11: Use biodiesel in government diesel vehicles
Action #12: B.C. Buys Green
Action #13: Support greener ports and marine vessels

Action #10, “Green vehicle fleets: we’re making it happen”, pertains to the Green Fleets BC initiative (a partnership with the Fraser Basin Council). The goal is to have Green Fleets BC be a key information hub for the latest on green technologies for private and public sector fleets, including taxis, emergency vehicles, delivery vans and commercial freight trucks. Initiatives include:

  • Green Trucking Technologies: Green Fleets BC will support demonstration projects on trucking technologies to help fleet managers make good choices for the best results.
  • BC BioFleet: Green Fleets BC is sponsoring demonstration projects on the use of biodiesel in the B.C. public sector and commercial vehicle fleets, through branding, website resources, training workshops, outreach and incentives.
  • Hybrid Experience Report: The report documents the real-life experiences and fuel efficiency results of fleets that use hybrid vehicles.
  • Green Fleets Network: Green Fleets BC is sponsoring a new network of fleet managers.
  • Canada’s Green Fuels Map: The Canada's Green Fuels Map provides an online lookup of both retail and key-lock stations, coast to coast.
  • Emissions calculator tool.

Time frame / duration

Launched in 2007
Website features information from 2007 to 2009, but should have funding from 2008 to 2011


Clean Nova Scotia - Fleetwiser Program

Funding (source and amount)

Funded by NRCan (ecoENERY for Fleets)

$ - N/A

Partners

Clean Nova Scotia (not-for-profit organization)

Target audience

Municipalities and utilities

Context (rationale and need for the program)

“Solutions are available today that greatly reduce vehicle fuel consumption and harmful emissions, what's needed is the will to make the change.”

Objective/goal

Fleetwiser seeks to create a cleaner, healthier environment by informing, enabling and inspiring Nova Scotian fleets to respect and consider the environment in all their choices.

Mechanisms/activities completed

Clean Nova Scotia’s Fleetwiser Pilot Project is working with four municipalities and one utility to improve vehicle fleet efficiency and reduce fuel consumption, GHG emissions, and business costs.

Participating fleet managers and operators receive free consultation, training and resources to produce measurable efficiency improvements. The program delivers fleet energy efficiency through education, technology, outreach, and driver awareness initiatives. Fleet baseline data, NRCan’s online Fleet Emissions Tool accounts, and individual fleet action plans are in development. The knowledge and tools gained from the project will be used to improve the fuel efficiency of other fleets that are interested.

Initiatives to date:

  • Nova Scotia municipal and utility fleet managers have participated in Fuel Management 101 workshops.
  • Fleet Efficiency Tool Kits have been developed for fleet managers, operators and technicians, and are accessible through Fleetwiser’s website, as are other key resources.
  • Strategies implemented by municipalities / utilities have included: anti-idling and efficient driving promotion and education, in-car training, preventive maintenance, and green procurement planning to optimize fleet equipment, technology and vehicles.
  • Customized vehicle-tracking systems and idle-reduction technologies are being tested on vehicles such as snow plows and salt trucks, bucket-lift trucks, buses, and light-duty vehicles.

Time frame / duration

From 2009 to 2011


PPRE Pilot Project

Funding (source and amount)

Environment Canada (through Industry Canada’s SPP)
$1.2 million for 2008 to 2011

Partners

FCM
MSC
CAP

Target audience

Municipalities
School buses – school bus operators and districts

Context (rationale and need for the program)

The PPRE complements CARA by addressing emissions from existing vehicles and engines that could remain on Canada’s roads for up to 20 years. It targeted on-road HDD vehicles such as trucks and school buses, and explores how best to voluntarily reduce their emissions.

Objective/goal

The objective is to reduce emissions from Canada’s on-road HDD municipal fleet. The project aims to increase knowledge, awareness and supporting tools for municipalities to reduce their emissions, by highlighting emission reduction potentials and identifying barriers to the widespread adoption of emission reduction strategies for fleet vehicles.

Mechanisms/activities completed

Environment Canada worked with the FCM to enable the following activities:

  • Establish partnerships with municipal fleet managers’ networks, federal and provincial agencies, environmental NGOs and private sector organizations.
  • Create a baseline of municipal heavy-duty fleets in Canada, including a fleet profile, an overview of best practices, and an identification of barriers to greening fleet initiatives.
  • Identify and create an online resource in both official languages for municipal fleet managers that will provide information on best practices available to achieve emission reductions.
  • Promote, in both official languages, the resources guide and other products and services.
  • Deliver education and training workshops in collaboration with partners, using resources identified to increase the capacity of municipal fleet managers to act.
  • Develop a business case for emission reduction initiatives in heavy-duty fleets that can be included in a resources guide.

In addition to the work completed in conjunction with FCM, Environment Canada worked with other Government of Canada ecoTRANSPORT funding programs (i.e., Transport Canada’s ecoFREIGHT, NRCan’s FleetSmart) to ensure they were aware of these activities for the on-road public HDD fleets. Furthermore, Environment Canada worked in conjunction with CAP and MSC to collect information about, and build capacity to address, emissions from school buses in select communities in Ontario.

Time frame / duration

From 2008 to 2011

Source

Evaluation Work Plan, February 2011

This analysis has demonstrated that all selected capacity-building programs have a similar temporal scope of two to four years, in line with the PPRE’s three-year time frame. In addition, all programs partnered with an NGO to deliver their activities. All target audiences for the programs included municipalities, although the Green Fleets BC initiative also targeted the commercial sector, whereas Fleetwiser worked with one utility and the PPRE’s scope included school bus operators and school districts.

Although none of the programs were found to have a specific objective to build the capacity of municipal fleet operators responsible for on-road HDD vehicles, the literature review demonstrated that there are multiple alternatives to building capacity in order to reduce vehicle emissions more generally, including the following:

  • Conducting audits of municipal fleets, which focus on identifying retrofit opportunities by conducting an on-site audit of a municipal fleet
  • Workshops, such as for fleet managers regarding fuel efficiency or how to green their heavy-duty vehicle fleet
  • Networking, which includes networking opportunities through joining various fleet manager networks
  • Developing guidelines and toolkits, which are provided to fleet managers and cover a range of topics ranging from best practices to emerging technologies
  • Links to other resources, primarily through the organizations’ websites
  • Testing of reduction technologies, which focuses on supporting demonstration projects and purchasing reduction technologies

Return to Table of Contents

Annex 3 Interviewees and Interview Guides

Interviewees

Internal:

1 Senior manager from Environment Canada
2 Staff from Environment Canada (HQ)
3 Staff from Environment Canada (Ontario Region)

External – Other Government Department:

1 Staff from NRCan

External – Partners/NGOs:

1 Manager from FCM
1 Staff from FCM
1 Staff from CAP
1 Staff from MSC

Interview Guides

Letter of Introduction

Date:
February XX, 2011
To:
Key Informant
Subject:
Evaluation of the Pilot Project on reducing emissions from vehicles and engines

Environment Canada (EC) is currently conducting an independent evaluation of the Pilot Project to reduce emissions from vehicles and engines (PPRE). You should have already received an email from program staff in EC requesting your participation in this study; along with a call from the Evaluation Team to schedule this interview. I am writing to confirm your scheduled interview for February xx, 2011 at xx am/pm. We very much appreciate your valuable contribution to this research.

The planned interview, to be conducted by a consultant from Stratos, will take around one hour and the information you provide will not be directly attributed to you, rather your input and candid response will help inform our evaluation by contributing to our formulation of conclusions and recommendations. Interviews can be conducted in either English or French according to your preference, and you will have the option to answer questions you are most comfortable with. If your schedule has changed or you are not available, we would appreciate if you could suggest another time that is more convenient or nominate another individual who could participate in this evaluation in your place.

Attached, you will find a series of questions that will serve as the basis for the scheduled interview.

If you have any comments or concerns, please feel free to contact Nicola Scahill, Transportation Division, Environment Canada, 819-997-9284 (nicola.scahill@ec.gc.ca), or Michael Gullo, Project Manager, Stratos, 613-241-1001x231 (mgullo@stratos-sts.com).

Interview Guide – Internal

Relevance

1)  What is your involvement and/or role with the PPRE? (Introduction)

2)  In your opinion, to what extent does the PPRE address an actual need? What ‘environmental need’ or gap did it address and how was this determined? (EQ1)

3)  Does the PPRE address recent/current federal priorities? If so, which ones and to what degree? (EQ2)

4)  How does the PPRE help meet EC’s current strategic outcomes and expected results? If so, which ones and to what degree? (EQ2)

5)  What role/responsibility is there for the Federal Government to:

  1. encourage voluntary reduction of emissions from Canada’s on road heavy duty diesel municipal fleet, and
  2. to work on collaborative technical studies with the US to reduce GHG emissions from on-road fleets in support of future regulatory actions? (EQ3)

6)  Can you identify other similar programs offered by the Government of Canada that compliment or duplicate the objectives or activities of the Pilot projects? (EQ1)

Project Design, Delivery and Effectiveness

7)  To what extent are the roles and responsibilities clear to all involved parties? (EQ5)

8)  How was the project designed to meet its expected results (e.g. what performance data is collected and reported)? How effective has the program been in achieving its expected results? Can you comment on whether the project did not achieve any of its expected results? (EQ 4 & 5)

9)  What have been the key outputs and results from the projects to-date? (e.g., number and population of municipalities implementing green fleet initiatives, number of technical studies done in collaboration with U.S. EPA, etc.) (EQ9)

10)  How has information been shared with other public fleet operations or other fleets that operate within a municipal context? (EQ7)

11)  What have been the main strengths, weaknesses, and lessons learned in implementing the projects – what worked best, what was critical to success, what were the main challenges? (EQ8)

12)  Do you foresee any potential barriers for expanding this project or replicating it a national scale? (EQ6)

13)  Have there been any unintended outcomes as a result of the pilot (positive or negative)? (EQ10)

Project Efficiency and Economy

14)  What measures have been implemented to enhance the efficiency of the PPRE (i.e., keep costs as low as possible) (EQ11)?

15)  Do you think that the investments made in the PPRE are a good use of public funds? Why? (EQ11 & 12)

16)  Are there alternative means or models to enhancing efficiency and economy with these kinds of projects? (EQ5 & 11)

Interview Guide – External

Relevance

1)  What was your involvement in project #X/Y? (Introduction)

2)  In your opinion, what ‘environment need’ does Project X/Y address? Why was this project necessary? (EQ1)

3)  In your opinion what is the appropriate role for the federal government to play with respect to reducing emissions from vehicles and engines?

Project Design, Delivery and Effectiveness

4)  What outputs or information has the project provided you with (e.g. technical studies, new information on retrofits, etc.)? Why is this information valuable and has it helped you contribute to other activities or initiatives? (e.g., implementation of green fleet initiatives, school bus retrofits, ongoing contributions to research, etc.) (EQ1, EQ9)

5)  What have been the main strengths, weaknesses, and lessons learned in implementing the projects – what worked best, what was critical to success, what were the main challenges? (EQ8)

6)  Do you foresee any potential barriers for expanding this project or replicating it at a national scale? (EQ6)

7)  Have there been any unintended outcomes as a result of the pilot (positive or negative)? (EQ10)

Project Efficiency and Economy

8)  Do you think that the investments made in this Pilot Project are a good use of public funds? Why? (EQ11 & 12)

9)  Are there alternative ways to implement projects such as this? What would be the pros and cons of these alternatives? (EQ5 & 11)

10)  Are there more effective or efficient ways to deliver the program, or obtain similar results? (EQ5 & 11)

11)  What would have been the impact if this Pilot Project did not exist? (EQ1)

Return to Table of Contents

Annex 4 Summary of Findings

Annex 4: Summary of Findings
Evaluation IssueAchievedProgress Made;
Attention Needed
Little Progress;
Priority for Attention
N/A
Relevance:
Continued need for the program
1.
Was there a need for the project? Is there a need for the project to continue?
   
Alignment with federal government priorities
2.
Is the project aligned with federal government priorities?
   
Consistency with federal roles
3.
Is the project consistent with federal roles and responsibilities?
   
Performance – Project Design and Delivery:
4.
Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?
   
5.
Is the program design appropriate for achieving expected program results?
   
6.
Are there any barriers or challenges that could affect the success of this project if it were to be implemented nationally or integrated into an existing national project?
   
7.
Have the results of the pilot project been communicated effectively? How could communications be improved?
   
8.
What are the best practices and lessons learned as a result of the project?
   
Performance – Effectiveness:
9.
To what extent have the intended outputs and outcomes been achieved as a result of the project?
   
10.
Have there been any unintended outcomes, either positive or negative?
   
Performance – Efficiency and Economy:
11.
Has the project undertaken its activities and delivered products (e.g., technical studies) in the most efficient manner?
   
12.
Has the project achieved, or is it on track to achieving, its intended outcomes in the most economical manner?
   

Return to Table of Contents

Annex 5 Table 10

Table 10 presents the expenditures of the Pilot Project on Reducing Emissions from Vehicles and Engines against original allocations from 2008-09 to 2010-11

Table 10: PPRE Allocations and Expenditures
 2008-092009-102010-11Total 2008-11
AllocatedSpentAllocatedSpentAllocatedSpent*AllocatedSpent
* Projected expenditures; in some cases figures were rounded by program staff.
** Includes costs associated with program travel, translation, and a co-op student.
*** Includes $75,000 that was reallocated to the first initiative.
Reducing Emissions from Canada’s On-Road HDDMunicipal Fleet
FTEs0.5 FTE0.5 FTEFTEFTEFTEFTE  
O&M Allocation$52,770 $73,009 $73,009 $198,788 
O&M Expenditure: 
Program Design $52,000   $52,000
Bronson Consulting  $25,000  $25,000
CAP   $25,000 $25,000
MSC  $42,000  $42,000
Other**  $6,000 $23,000 $29,000
Evaluation   $25,000 $25,000
Total O&M$198,788$198,000
G&C Allocation  $148,664 $223,664*** $372,328 
G&C Expenditure: 
FCM   $32,000 $140,000 $172,000
MSC     $83,664 $83,664
Total G&C $372,328$255,664

Return to Table of Contents

Date modified: