Skip booklet index and go to page content

Evaluation of the National Vehicle Scrappage Program

Annex 3 Evaluation Issues and Questions

Final Report

July 12, 2011

Relevance

Was the Vehicle Scrappage Program consistent with, and did it contribute to, the federal government priorities, and did it address actual needs?

Question 1. a)

Did the Vehicle Scrappage Program respond to an identified need? What was that need?

Indicators
  • Demonstration of societal/environmental need
  • Views on connection of program objectives with societal/environmental need
Data Sources/Methods
  • Document review
  • KI interviews
  • Case studies (recycling code of practice)
TB Policy Issue Addressed
  • Issue #1: Continued Need for Program

Top of Page

Question 1. b)

Is there a continued need for the program?

Indicators
  • Degree to which gaps would exist in addressing societal/environmental need in absence of the program
  • Views on whether there is a continued need for the Vehicle Scrappage Program
Data Sources/Methods
  • Document review
  • Literature review (including statistics on age of vehicles in use)
  • KI interviews
  • Case studies (recycling code of practice)
TB Policy Issue Addressed
  • Issue #1: Continued Need for Program

Top of Page

Question 2.

Did the program complement or duplicate other related federal and/or provincial government programs (i.e., Transport Canada programs, provincial emission testing programs, etc.)?

Indicators
  • Presence/absence of other government programs (federal and provincial) that complement or duplicate the objectives of the program
  • Views on whether the program complemented or duplicated other related programs
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #1: Continued Need for Program

Top of Page

Question 3.

Was the Vehicle Scrappage Program aligned with departmental and federal government priorities?

Indicators
  • Extent to which the program’s objectives corresponded to recent/current federal government priorities (e.g., CAA objectives)
  • Extent to which the program’s objectives aligned with recent/current departmental strategic outcomes
  • Views on the alignment of the program’s objectives with recent/current departmental and federal government priorities
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #2: Alignment with Federal Government Priorities

Top of Page

Question 4.

Was the Vehicle Scrappage Program consistent with the Department’s mandate and with federal roles and responsibilities?

Indicators
  • Extent to which program mandate was aligned with departmental mandate and federal government jurisdiction
  • Views on the appropriateness of departmental and federal involvement
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #3: Alignment with Federal Roles and Responsibilities

Top of Page

Performance

Has the Vehicle Scrappage Program achieved its intended outcomes? Were the most appropriate, efficient and economic means used to achieve outcomes?

Top of Page

Question 5.

Were the objectives of the program realistic and adequately defined?

Indicators
  • Demonstration that the Vehicle Scrappage Program had realistic objectives
  • Degree to which the program’s objectives are clearly understood by key stakeholders
  • Views on the viability and clarity of program objectives
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 6. a)

Was the design of the Vehicle Scrappage Program appropriate for achieving expected program results?

Indicators
  • Plausibility of link between program activities, outputs and intended outcomes
  • Extent to which program processes and governance structure, including roles, responsibilities and accountabilities, were clearly defined and understood
  • Views on appropriateness of program activities, processes and governance structure
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 6. b)

Was the funding of the program commensurate with its objectives and the needs identified?

Indicators
  • Evidence of the degree to which program resources and capacity were commensurate with expected program results
  • Views on adequacy of program funding
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 7. a)

Were participants satisfied with the administration of the program by both the Department and the not-for-profit organizations?

Indicators
  • Level of participants’ satisfaction with administration of the program by the Department and the NFPs
Data Sources/Methods
  • Secondary data analysis (tracking research on program participants)
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 7. b)

Were participants satisfied with the incentives provided to reward them for scrapping their vehicle (type and value)?

Indicators
  • Level of satisfaction of participants with the incentives provided through the program
Data Sources/Methods
  • Secondary data analysis (tracking research on program participants)
  • Case studies
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 8.

Were the achievements of the Vehicle Scrappage Program effectively and efficiently captured and disseminated to appropriate audiences (i.e., key stakeholders and the general public)?

Indicators
  • Degree to which program activities and results were effectively communicated to key stakeholders (including provision of input into CAA Horizontal Management Accountability and Reporting Framework)
  • Availability of information on program results for the Canadian public
  • Views on extent to which program achievements were effectively and efficiently captured and shared with key stakeholders and general public
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 9.

Have appropriate performance data been collected, captured and safeguarded? If so, has this information been used to inform senior management / decision makers?

Indicators
  • Existence of an effective performance measurement strategy
  • Extent to which performance data are reliable, timely and relevant
  • Extent to which performance data informed/supported EC’s decision-making processes
  • Views on strengths and weaknesses of the Vehicle Scrappage Performance Monitoring Framework
Data Sources/Methods
  • Document review
  • Secondary data analysis (program performance data)
  • KI interviews
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 10. a)

What are the lessons learned (both strengths and weaknesses) from the program?

Indicators
  • Views about the program’s strengths, best practices and weaknesses (what worked well, what has not worked well, and what could have been done better)
Data Sources/Methods
  • Document review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 10. b)

Are there follow-up activities or programs that could build on the success achieved by this program?

Indicators
  • Extent to which program activities could lead to further future opportunities to meet the overall CAA objectives or other environmental priorities
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 11.

To what extent have the intended outcomes been achieved as a result of the Vehicle Scrappage Program?

Question 11. a) Short-term outcomes

Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road

Indicators
  • Number of provinces and territories (agreed upon in approved operating plan) in which program was available
  • Number of provinces and territories (agreed upon in approved operating plan) in which trained recyclers were available
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (program database)
  • Document review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. b) Short-term outcomes

Increased awareness by the public of the program and the environmental impacts of older vehicles

Indicators
  • Change in level of awareness among Canadians (the general population and owners of older vehicles) about the environmental impacts of older vehicles
  • Change in attitude toward the use of older vehicles
  • Change in level of awareness among Canadians about the program
  • Degree to which the public believed the program was a good idea
  • Number of requests for information about incentives received via the website and/or 1-800 O-Canada (Service Canada)
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (tracking research on public)
  • Document review
  • KI interviews
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. c) Short-term outcomes

Increased awareness by recyclers of environmentally safe recycling practices

Indicators
  • Recyclers’ level of acceptance of and attitude toward the recycling code of practice
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (Vehicle Recyclers Survey)
  • KI interviews
  • Document review
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. d) Medium-term outcomes

Fewer older, high-polluting personal vehicles on the road

Indicators
  • Number of older high-polluting personal vehicles scrapped as part of the program
  • Reported influence of the program on participants’ decision to recycle their vehicle
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (program database and tracking research on program participants)
  • Document review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. e) Medium-term outcomes

More sustainable transportation modes being used for personal transportation

Indicators
  • Uptake of incentives that do not involve the ownership of another vehicle
  • Use of sustainable transportation during the incentive life (e.g., one year of free transit passes)
  • Potential use of sustainable transportation beyond incentive life
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (program database and tracking research on program participants)
  • Document review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. f) Medium-term outcomes

Environmentally safe practices in use for recycling of vehicles scrapped through the program

Indicators
  • Level of adherence with code of practice by vehicle recyclers
  • Level of support from provincial recycling associations and provincial governments for code of practice
  • Extent of adoption of code of practice by provinces and other stakeholders
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • KI interviews
  • Document review
  • Literature review
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. g) Long-term outcomes

Air pollutants are reduced

Indicators
  • Calculated reductions in smog-forming emissions
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (program database)
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. h) Long-term outcomes

GHG emissions are reduced

Indicators
  • Calculated reductions in GHG emissions
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (program database)
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. i) Long-term outcomes

Release of harmful and toxic substances into the environment is prevented

Indicators
  • Number of vehicles recycled according to code of practice
  • Views on extent to which outcome has been achieved
Data Sources/Methods
  • Secondary data analysis (program database)
  • Document review
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 12.

Did this program lead people to make use of other related federal and provincial government programs or vice-versa?

Indicators
  • Level of involvement of program participants in the use of other related programs
  • Views on extent to which outcome has occurred
Data Sources/Methods
  • Secondary data analysis (tracking research on program participants)
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 13.

Were there any external factors outside of the Vehicle Scrappage Program that influenced (positively or negatively) the success of the program?

Indicators
  • Evidence of factors outside the program which have influenced the achievement of intended outcomes
  • Views on whether there were any external factors that helped or hindered the program’s success
Data Sources/Methods
  • Document review
  • Literature review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 14.

Were there any unintended positive or negative outcomes of the Vehicle Scrappage Program?

Indicators
  • Presence or absence of unintended outcomes
  • Views on whether unintended outcomes occurred as a result of the program
Data Sources/Methods
  • Document review
  • KI interviews
  • Case studies
TB Policy Issue Addressed
  • Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 15.

  1. Did the Vehicle Scrappage Program undertake its activities and deliver its products in the most efficient manner?
  2. Was federal funding leveraged to the extent planned?
  3. How could the efficiency of the program be improved? program’s activities be improved?
  4. Are there alternative, more efficient ways of delivering the program?
Indicators
  • Analysis of actual program operational costs in relation to the production of outputs
  • Analysis of program administrative costs in relation to total program costs
  • Views on whether the cost of producing program outputs was as low as possible
  • Amount of financial commitments from incentive partners
  • Amount of financial commitments and other support from provinces and territories toward the program’s delivery
  • Number of partnerships achieved by national NFP with incentive partners that supported the program’s goals
  • Views on how the efficiency of program activities could be improved
  • Comparison of program activities and products delivered by other similar programs, including U.S. Cash for Clunkers
  • Views on whether there are alternative, more efficient ways of delivering program activities and outputs
Data Sources/Methods
  • Document review
  • Financial analysis
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #5: Demonstration of Efficiency and Economy [Efficiency]

Top of Page

Question 16.

Did the Vehicle Scrappage Program achieve its intended outcomes in the most economical manner?

Indicators
  • Extent to which the program’s intended outcomes have been achieved at the least possible program cost
  • Views on whether good value was obtained with respect to the use of public funds
  • Evidence of / views on whether there are alternative program models that would achieve the same expected outcomes at a lower cost
Data Sources/Methods
  • Document review
  • Financial analysis
  • Literature review
  • KI interviews
TB Policy Issue Addressed
  • Issue #5: Demonstration of Efficiency and Economy [Economy]
Date modified: