Formative Evaluation of the Federal Contaminated Sites Action Plan

Final Evaluation Report

| ToC | Previous | Next |


1.0 Introduction

1.1 Background

The Federal Contaminated Sites Action Plan (FCSAP) is a cost-shared program (between the FCSAP fund and Custodians of federal contaminated sites) with the objective to support federal Custodians in addressing contaminated sites for which they are responsible.  The FCSAP Secretariat, housed at Environment Canada (EC), co-administers the program with the Treasury Board Secretariat (TBS).  Environment Canada, Fisheries and Oceans Canada (DFO) and Health Canada (HC) provide expert support to Custodians on a broad scope of issues related to potential ecological and human health risks posed by the sites; additionally, Public Works and Government Services Canada (PWGSC) provides support and advice to Custodians on project management and innovative technologies. Reporting on the effectiveness of the program is one of the key roles of the FCSAP Secretariat.

In 1996, the federal Office of the Auditor General estimated that there were approximately 5000 contaminated federal sites in Canada.3 By 2004, this number had grown to 6000 and in Budget 2004 the federal government committed $3.5 billion to address federal contaminated sites, an amount equivalent to the related financial liability reported in the Public Accounts at that time. An additional amount of $500 million was also announced in the 2004 Federal Budget for shared responsibility sites.  

1.1.1 Objectives of FCSAP

Stewardship for federal real property, including responsibility and accountability for managing contaminated sites, rests with Custodians. The FCSAP complements ongoing custodial activities on contaminated sites and priority is given to those sites that present the highest risk to human health and the environment. 

Under the FCSAP, a consistent management regime has been established to address federal contaminated sites within a 15-year timeframe. The specific FCSAP program objectives include: 

1.1.2 Delivery Approach4 

This section describes the contributions of the FCSAP Secretariat, Expert Support departments and Custodians in the delivery of the FCSAP. It should be noted that some departments have multiple roles (e.g., Expert Support and Custodian).

FCSAP Secretariat

The FCSAP Secretariat at EC is responsible for the administration of the program, including leading and coordinating its delivery, coordinating the review of proposals and managing the project selection process.  The FCSAP Secretariat also tracks funding requests and project expenditures and develops funding allocation proposals for approval by the FCS Steering Committee.  The FCSAP Secretariat develops the procedures to ensure interdepartmental consistency in program implementation and also provides clerical and administrative services to the FCS Steering Committee and the CSMWG.  

The TBS (Real Property and Materiel Policy Division) ensures consistency with TB policies on the management of federal real property, including federal contaminated sites, administers and maintains the FCSI and assists the FCSAP Secretariat with the monitoring of and reporting on government-wide progress in addressing federal contaminated sites funded under the FCSAP.

One database, the Interdepartmental Data Exchange Application (IDEA), is maintained at Environment Canada where the FCSAP Secretariat is housed, while the TBS maintains the other database, the Federal Contaminated Sites Inventory (FCSI).  Environment Canada developed IDEA as a project-specific database, in which a record is maintained of all sites within a given project.  Its purpose is to facilitate the exchange of information between the FCSAP Secretariat, Custodians of federal contaminated sites (departments, agencies, and consolidated Crown corporations), and the Expert Support departments (HC, EC, DFO, and PWGSC).  On IDEA, Custodians submit new project funding requests, update their previously approved submissions, and report annually on their FCSAP funded projects.  As required by Treasury Board policy, TBS has hosted the FCSI since 2000. The FCSI provides Custodians with access to a central inventory in which they can maintain a record of all contaminated sites (including those that are not funded under FCSAP) for which the Custodian is responsible.  

Expert Support5

The FCSAP Secretariat is assisted by four Expert Support departments (HC, EC, DFO and PWGSC) whose main role is to provide technical advice, training and guidance to Custodians. 

EC, HC and DFO provide advice to the FCSAP Secretariat to develop and promote best practices so that Custodians adopt a consistent national approach to human health and ecological risk assessments.  These three Expert Support departments also provide project-specific advice to allow Custodians to make a final decision on project activities; develop guidance materials and training; contribute to the understanding and management of human health and ecological issues; liaise with their provincial and territorial counterparts on behalf of Custodians as appropriate; participate in interdepartmental regional working groups; advise on risk management approaches, risk communication strategies and public outreach; and act as sources of expert knowledge and information related to environmental assessment.  These three departments also carry out their respective mandates related to regulatory compliance.

PWGSC provides expert support to the program through the development of project management tools, the dissemination of information on innovative technologies and technologies used in individual projects, and liaison with industry.  PWGSC also coordinates forecasts of project requirements and procurement opportunities to support linkages to other federal priorities and provides information to industry.  Additionally, PWGSC administers an awards and recognition program for best practices.

All Expert Support departments (EC, DFO, HC and PWGSC) are required to prepare and submit to the FCSAP Secretariat an annual report on their activities and expenditures under the FCSAP.  These reports are made available for review by the CSMWG in the fall.

Other departments provide support to the program within their existing mandates.  Human Resources and Social Development Canada provides support to Custodians on opportunities to build capacity of the environmental industries sector to manage risks at federal contaminated sites; and on opportunities to develop synergies with the government’s skills development and employment agenda.  Industry Canada supports Custodians, most specifically PWGSC, in facilitating familiarization and collaboration between contaminated site Custodians and remediation technology vendors, most particularly the introduction and advancement of innovative technologies.

Custodians

To date sixteen Custodians6 have accessed FCSAP funding.  Custodians seeking resources under the FCSAP program must:

1.1.3 Governance Structure

There are multiple components that interact within the governance structure for the FCSAP. The main components are described briefly below.

Federal Contaminated Sites Steering Committee: EC and TBS co-chair the ADM-level Federal Contaminated Sites Steering Committee (the FCS Steering Committee), which oversees the implementation of the FCSAP program.  The FCS Steering Committee is responsible for setting project priorities, monitoring progress, and providing recommendations on the funding of FCSAP projects for Treasury Board approval.  The mandate of the FCS Steering Committee is to review and approve FCSAP project proposals, as recommended by the CSMWG, and recommend funding allocations to Treasury Board, in accordance with TB-approved FCSAP selection criteria. The FCS Steering Committee will ensure that allocated FCSAP funds are used to identify, manage, and remediate federal contaminated sites in a manner that is consistent with the TB Policy on Management of Real Property.

Contaminated Sites Management Working Group (CSMWG): EC and DND currently co-chair the CSMWG, a working-level committee representing all Custodians and the four Expert Support departments. The CSMWG was established in 1995 to investigate and propose a common federal approach to the management of contaminated sites under federal custody and related issues.  The CSMWG plays an ongoing role in the development and implementation of the FCSAP program by contributing to the development of procedures, tools and guidance, and other key program outputs; reviewing proposed program funding plans; and making recommendations on project funding to the Steering Committee.  The CSMWG meets approximately eight to ten times per year. The CSMWG also establishes subcommittees and working groups, as appropriate, to address scientific and technical issues, and provide support to Custodians on opportunities related to linkages to other socioeconomic outcomes. These include skills development, training and employment of Canadians, and technological development in the environment industry.

FCSAP Interdepartmental Regional Working Groups (IRWGs): IRWGs are in place in EC regions to advise Custodians on the management of contaminated sites. They are comprised of Expert Support members and Custodians’ contaminated sites regional staff. The IRWGs provide Custodians with training and access to the advice of Expert Support departments on compliance, health and ecological risks/impacts of contaminated sites, and risk-assessment approaches as well as advice on the development of remediation/risk-management plans for their sites, with priority given to those projects funded under the FCSAP program.

1.1.4 Funding Allocation

FCSAP funding is to be used to reduce risk and liability, investigate potential contaminated sites, and remediate and risk-manage those requiring attention.  Program funds can be used for site assessment, remediation/risk management, and care and maintenance activities to prevent an increase in federal financial liability related to these sites. Approximately 90% of the annual funding amount is directed to projects and the remaining 10% is allocated to expert support, management, and administration.

FCSAP operates on a cost-shared basis with Custodians. For projects, the FCSAP will fund the first $10 million of project costs at a cost-share ratio of 80/20 (FCSAP funding/Custodians). Once total project costs exceed $10 million, the cost share ratio becomes 90/10.  Subject to Treasury Board approval, some exceptional sites exceeding $90 million may be funded at 100% of project costs.

Additionally, the program will fund site assessments at an 80/20 cost-share ratio to a maximum from the fund of $25 million per year, which can be adjusted only at the discretion of the Treasury Board.  

1.2 Scope and Objectives of the Evaluation

From January to April 2007, the FCSAP Program Evaluation Working Group and consultants worked to revise the FCSAP logic model and to develop the most appropriate overall approach for the evaluation of FCSAP.  The general consensus was that the formative evaluation approach would take into account the following:

1.2.1 Evaluation Objectives

The formative evaluation is intended to provide findings mid-way through the initiative.  The primary objective of the formative evaluation is to provide information on the appropriateness of design and delivery of the FCSAP at a stage in the program cycle when changes or adaptations can be made to optimize the likelihood of success in achieving expected outcomes.  As a second objective, the formative evaluation will examine the preliminary success achieved to date with respect to immediate and intermediate outcomes, and potential areas to explore further in the summative evaluation.  

1.2.2 Evaluation Scope

The scope of the formative evaluation includes all federal contaminated sites and projects that have been identified, assessed and/or worked on during the two fiscal years of 2005–2006 and 2006–2007.7 This scope was applied to all evaluation methods including the examination of existing administrative databases such as the FCSI and the IDEA.  

The formative evaluation issues will focus on the FCSAP Secretariat, all Expert Support departments, and a sample of Custodians. Four of the sixteen Custodians presently receiving funding as Custodians were included in the scope of the formative evaluation.  These four departments were chosen to ensure an adequate level of coverage of project types (assessment, remediation/risk management and care and maintenance) and size (from under 50K to over $10M), and collectively account for 86% of the number of FCSAP funded sites.  

The following departments participated in the evaluation:

1.2.3 Evaluation Issues and Questions

As outlined in the statement of work derived from the evaluation framework, there were fourteen key evaluation questions addressed in the formative evaluation.  One main issue addressed was the design and delivery of the FCSAP, which has ten associated evaluation questions.  As well, there was some emphasis on the issue of FCSAP success with four associated evaluation questions.  The specific questions addressed by the evaluation were:

Issue: Design and Delivery of the FCSAP

Evaluation Questions

  1. Is the FCSAP governance model comprehensive, clear and appropriate for a horizontal initiative? 
  2. Are roles and responsibilities clear and appropriate for a horizontal initiative such as FCSAP? Who is accountable for the program? Is there an appropriate horizontal accountability framework?
  3. To what extent has FCSAP been implemented as originally planned?  If not implemented as planned, what was the rationale for different implementation?
  4. Are decision-making processes in place to allow for the highest areas of importance to be reflected in the allocation of resources (priorities)?
  5. Is risk appropriately addressed?
  6. To what extent are FCSAP activities and outputs appropriately linked to shared outcomes and the overall horizontal mandate?
  7. Are all components of the FCSAP appropriately resourced?
  8. Is the funding process supportive of FCSAP objectives?
  9. Is the performance measurement sufficient?
  10. To what extent has the FCSAP produced key outputs as planned? (see logic model for specific planned outputs) 

Issue:  FCSAP Success

Evaluation Questions

  1. To what extent has the FCSAP achieved immediate outcomes?
    1. Increase in the number of Class 1 & 2 federal contaminated sites for which remediation/risk management plans are being developed or have been developed and/or implemented.
    2. Greater reliability of FCSI and financial liability estimates.
    3. More accurate picture of federal contaminated sites, ranked according to human health and ecological risks.
    4. Increased availability and awareness of nationally consistent management and communications tools to address highest risk sites.
  2. What is the likelihood that FCSAP will be able to achieve the anticipated intermediate outcomes within the timeframe indicated? What will be the anticipated challenges/barriers to achieving and/or demonstrating these intermediate outcomes?(Refer to logic model for specific intermediate outcomes)
  3. At this stage, what external factors have contributed to, or detracted from, FCSAP’s level of success? 
  4. Have there been any unanticipated results, either positive or negative, that can be attributed to the program?
    1. If so, how were they addressed?
    2. If so, to what extent are these results related to the FCSAP being a horizontal initiative?

 


| ToC | Previous | Next |


3 Please refer to the 1996 Office of the Auditor General Report for further details.

4 Appendix C presents the FCSAP Logic Model (as of 2007), which presents the key activities and outputs, for each of the three components of the FCSAP, namely, the FCSAP Secretariat, Expert Support and Custodians, as well as shared immediate, intermediate and long-term outcomes.

5 FCSAP RMAF (2006)

6 The sixteen Custodians are: Agriculture and Agri-Food Canada (AAFC); Canadian Food Inspection Agency (CFIA); Canada Border Services Agency (CBSA); Correctional Services Canada (CSC); DFO; DND; EC; HC: INAC; Jacques Cartier and Champlain Bridge Incorporated; National Capital Commission (NCC); Natural Resources Canada (NRCan); Parks Canada Agency (PC); PWGSC; Royal Canadian Mounted Police (RCMP); and Transport Canada (TC).

7 Note: The scope includes projects that commenced prior to 2005.