Home >
Proactive Disclosure >
Audits and Evaluations >
A&E Reports >
2011-2012 >
Evaluation of the National Vehicle Scrappage Program
Evaluation of the National Vehicle Scrappage Program
Final Report
July 12, 2011
Previous page | Table of Contents | Next page
3.0 Evaluation Design
3.1 Purpose and Scope
In order to meet a TB commitment, an evaluation of the NVSP was conducted in 2010-2011. This evaluation was part of EC’s Risk-Based Evaluation Plan for 2010-2011 to 2014-2015.
The evaluation’s objective was to assess the relevance and performance of the NVSP. The evaluation focused primarily on the four-year time frame from 2007-2008 to 2010-2011, but also encompassed an examination of program data in 2011-2012 on the total number of vehicles retired and final planned program expenditures.
3.2 Evaluation Approach and Methodology
The evaluation examined the following two key issues, in accordance with the 2009 TB Policy on Evaluation:
- Relevance: the extent to which the program addressed a continued need, was aligned with federal government priorities, and was aligned with federal roles and responsibilities
- Performance: the extent to which the program achieved its expected outcomes, and demonstrated efficiency and economy
A total of 16 evaluation questions supporting these two broad issues were addressed in the evaluation. These evaluation questions, along with the associated indicators, data sources and data collection methods, are listed in Annex 3. The methodological approaches are briefly described below.
Top of Page
Literature and Document Review
A literature and document review was conducted to address most of the evaluation questions. The evaluation team, in cooperation with program representatives, began the process by selecting key documents and literature to be reviewed. The team conducted additional searches via online databases to complete the information, and reviewed documentation about similar programs offered in other jurisdictions within Canada and abroad in order to compare the features of each program and assess their strengths and weaknesses. The full list of documents appears in Annex 1.
The documentation was reviewed using a grid (based on the evaluation issues) that summarized the source and reliability of the information. After completing the grid, the evaluation team summarized the findings from the documentation per evaluation issue in a separate matrix. This summary served as a base for the draft report.
Top of Page
Secondary Data Analysis
The evaluation team also reviewed program data to address the evaluation questions. This included financial data from the program and performance data from SPARC, a database to track RYR vehicles and incentives.
The data were reviewed to obtain descriptive information about the program in order to establish its profile, and to gather data in order to address the evaluation issues. Tables were developed to describe the financial aspects of the program—in particular, the financial information was analyzed to assess the program’s cost efficiency compared to similar programs in Canada (ecoAUTO rebate) and abroad (Cash for Clunkers in the United States). The SPARC database was used to establish the reductions in smog, which were calculated based on EC and Natural Resources Canada emissions information.
Top of Page
Key Informant Interviews
A series of 20 in-depth, one-on-one key informant (KI) interviews was conducted using an interview guide, which included open-ended questions. The distribution of respondents to these interviews is outlined in Table 3 below:
Table 3: Distribution of Key Informant Interviews
| Category |
Completed Interviews |
Population of Stakeholders |
| EC program management and staff |
4 |
5 |
| NFP delivery organizations |
12 |
30 |
| Vehicle recycler associations |
2 |
2 |
| Provincial governments |
2 |
3 |
| Total |
20 |
40 |
The interviews were used to address most evaluation issues and were conducted in the respondents’ official language of choice. Interviews were conducted in person in the National Capital Region, and by phone for other areas. A visit to the Toronto area was also carried out, for in-person interviews with SI representatives and other stakeholders. Interviews were conducted using tailored guides, presented under separate cover in a technical appendix.
The results of the interviews were entered into a matrix, organized by evaluation issue for analytical purposes. After an overall analysis, the evaluation team summarized the results for the purposes of this report.
Top of Page
Case Studies
Certain aspects of the program were assessed as separate case studies. This allowed the evaluation team to conduct interviews and review documents focused on key aspects of the delivery of the program. The three cases are described below.
- Vehicle Recycling Code of Practice
As part of the NVSP, EC dedicated funding to help the automotive recycling sector develop a recycling code of practice. The code was intended to help recyclers improve their environmental performance and profitability while delivering valuable service to society. It was expected that the code would create a legacy of good environmental performance for the automotive recycling sector after the program sunsets. Seven individuals were interviewed and various documents were reviewed, including audit reports.
- Working with Partners to Provide Non-cash Incentives
In exchange for their vehicle, Canadians who qualified for the program could choose a reward from a suite of optional incentives. These varied from province to province, and included free transit passes, discounts on bicycles or e-bikes, memberships in car-sharing programs, and rebates on new or newer vehicles. SI and the NFPs arranged these higher-value non-cash incentives by negotiating and/or leveraging the $300 base incentive with local and national partners. Six individuals were interviewed, including NFPs and partners. Descriptive information from online sources was also reviewed.
- Communications
The scope and complexity of RYR provided many communications challenges (and opportunities) for EC and the program delivery partners. On a functional level, communications issues touched all aspects of the program and were at multiple levels, including internal and external communications (e.g., communication of information, advertising, service standards, bilingualism, federal funding). The case study was based on four interviews and a review of program documentation and statistics, including general population surveys, user surveys and outreach materials.
Overall, interviews and documentation were used to gain a better understanding of the program’s performance on each case study topic, as well as challenges, best practices and lessons learned. Interviews were conducted using guides tailored for each case study. The key results of the case studies are highlighted in the body of this report under the relevant evaluation questions. The case study guides and case reports are presented under separate cover in a technical appendix.
Top of Page
3.3 Limitations
As with any other evaluation, this study involved a number of limitations, including the following:
- Although the program’s performance was well monitored, certain aspects of the evaluation depended on information that was not documented (it is not unusual for some information to not be documented). To supplement the available performance data and documentation, the evaluation team relied on KI interviews to address a number of evaluation questions. KI interviews potentially provide in-depth responses, yet are also limited by a number of factors, including respondent bias and memory gaps. The KI interviews also covered only a selection of respondents, not the entire population of stakeholders involved in the program. The evaluation team is confident, however, that an appropriate mix of respondents was interviewed (i.e., an appropriate balance between knowledgeable respondents representing each key stakeholder group and respondents capable of providing an objective view of the program). The interpretation of interview findings also considered the other lines of evidence, including program documentation as well as literature and program data, all of which were extensive. Integrating the results from multiple lines of evidence helped to ensure the validity of the evaluation findings and conclusions.
- Although there are excellent data on the number of vehicles retired, there remain data gaps on the environmental impact of these removals. To assess the reductions in NOx, VOCs and other pollutants, the evaluation team used various assumptions, including assumptions based on Statistics Canada data (i.e., estimates of kilometres driven by typical vehicles) and the specifications of the retired vehicles and how they were replaced. The team based its assumptions on the best information available, but it is likely that the environmental impact is different than what is reported in this evaluation, as there was no assessment of the environmental performance of each vehicle upon its retirement. The evaluation team remained conservative in its assumptions, however, to help ensure that the reductions in pollutants were not overestimated.
Previous page | Table of Contents | Next page