Evaluation of the National Vehicle Scrappage Program

Final Report

July 12, 2011

Previous page | Table of Contents | Next page

3.0 Evaluation Design

3.1 Purpose and Scope

In order to meet a TB commitment, an evaluation of the NVSP was conducted in 2010-2011. This evaluation was part of EC’s Risk-Based Evaluation Plan for 2010-2011 to 2014-2015.

The evaluation’s objective was to assess the relevance and performance of the NVSP. The evaluation focused primarily on the four-year time frame from 2007-2008 to 2010-2011, but also encompassed an examination of program data in 2011-2012 on the total number of vehicles retired and final planned program expenditures.

3.2 Evaluation Approach and Methodology

The evaluation examined the following two key issues, in accordance with the 2009 TB Policy on Evaluation:

A total of 16 evaluation questions supporting these two broad issues were addressed in the evaluation. These evaluation questions, along with the associated indicators, data sources and data collection methods, are listed in Annex 3. The methodological approaches are briefly described below.

Top of Page

Literature and Document Review

A literature and document review was conducted to address most of the evaluation questions. The evaluation team, in cooperation with program representatives, began the process by selecting key documents and literature to be reviewed. The team conducted additional searches via online databases to complete the information, and reviewed documentation about similar programs offered in other jurisdictions within Canada and abroad in order to compare the features of each program and assess their strengths and weaknesses. The full list of documents appears in Annex 1.

The documentation was reviewed using a grid (based on the evaluation issues) that summarized the source and reliability of the information. After completing the grid, the evaluation team summarized the findings from the documentation per evaluation issue in a separate matrix. This summary served as a base for the draft report.

Top of Page

Secondary Data Analysis

The evaluation team also reviewed program data to address the evaluation questions. This included financial data from the program and performance data from SPARC, a database to track RYR vehicles and incentives.

The data were reviewed to obtain descriptive information about the program in order to establish its profile, and to gather data in order to address the evaluation issues. Tables were developed to describe the financial aspects of the program—in particular, the financial information was analyzed to assess the program’s cost efficiency compared to similar programs in Canada (ecoAUTO rebate) and abroad (Cash for Clunkers in the United States). The SPARC database was used to establish the reductions in smog, which were calculated based on EC and Natural Resources Canada emissions information.

Top of Page

Key Informant Interviews

A series of 20 in-depth, one-on-one key informant (KI) interviews was conducted using an interview guide, which included open-ended questions. The distribution of respondents to these interviews is outlined in Table 3 below:

Table 3: Distribution of Key Informant Interviews
Category Completed Interviews Population of Stakeholders
EC program management and staff 4 5
NFP delivery organizations 12 30
Vehicle recycler associations 2 2
Provincial governments 2 3
Total 20 40

The interviews were used to address most evaluation issues and were conducted in the respondents’ official language of choice. Interviews were conducted in person in the National Capital Region, and by phone for other areas. A visit to the Toronto area was also carried out, for in-person interviews with SI representatives and other stakeholders. Interviews were conducted using tailored guides, presented under separate cover in a technical appendix.

The results of the interviews were entered into a matrix, organized by evaluation issue for analytical purposes. After an overall analysis, the evaluation team summarized the results for the purposes of this report.

Top of Page

Case Studies

Certain aspects of the program were assessed as separate case studies. This allowed the evaluation team to conduct interviews and review documents focused on key aspects of the delivery of the program. The three cases are described below.

  1. Vehicle Recycling Code of Practice

    As part of the NVSP, EC dedicated funding to help the automotive recycling sector develop a recycling code of practice. The code was intended to help recyclers improve their environmental performance and profitability while delivering valuable service to society. It was expected that the code would create a legacy of good environmental performance for the automotive recycling sector after the program sunsets. Seven individuals were interviewed and various documents were reviewed, including audit reports.

  2. Working with Partners to Provide Non-cash Incentives

    In exchange for their vehicle, Canadians who qualified for the program could choose a reward from a suite of optional incentives. These varied from province to province, and included free transit passes, discounts on bicycles or e-bikes, memberships in car-sharing programs, and rebates on new or newer vehicles. SI and the NFPs arranged these higher-value non-cash incentives by negotiating and/or leveraging the $300 base incentive with local and national partners. Six individuals were interviewed, including NFPs and partners. Descriptive information from online sources was also reviewed.

  3. Communications

    The scope and complexity of RYR provided many communications challenges (and opportunities) for EC and the program delivery partners. On a functional level, communications issues touched all aspects of the program and were at multiple levels, including internal and external communications (e.g., communication of information, advertising, service standards, bilingualism, federal funding). The case study was based on four interviews and a review of program documentation and statistics, including general population surveys, user surveys and outreach materials.

Overall, interviews and documentation were used to gain a better understanding of the program’s performance on each case study topic, as well as challenges, best practices and lessons learned. Interviews were conducted using guides tailored for each case study. The key results of the case studies are highlighted in the body of this report under the relevant evaluation questions. The case study guides and case reports are presented under separate cover in a technical appendix.

Top of Page

3.3 Limitations

As with any other evaluation, this study involved a number of limitations, including the following:


Previous page | Table of Contents | Next page