This report presents the findings, conclusions, and recommendations for the Formative Evaluation of the Federal Contaminated Sites Action Plan (FCSAP) conducted from March to September 2008. FCSAP is a cost-shared program that assists federal departments, agencies, and consolidated Crown corporations to address contaminated sites for which they are responsible. The overall objective of FCSAP is to assess, remediate, or risk manage the highest-risk federal contaminated sites, and eliminate (or reduce as much as possible) the related federal liability.
With the program currently in the fourth year of its initial five-year funding period (2005–2006 to 2009–2010), the formative evaluation provides an opportunity to assess the appropriateness of the design and delivery of FCSAP, and to determine how well the program is operating and whether changes will be required to ensure that the program achieves its objectives. In addition, the program is at a stage where one would expect to observe progress being made toward achieving immediate outcomes.1
The scope of the evaluation included the FCSAP Secretariat, Steering Committee, Expert Support departments (Fisheries and Oceans Canada [DFO], Environment Canada [EC], Health Canada [HC], and Public Works and Government Services Canada [PWGSC]), and four Custodians (Transport Canada [TC], National Defence [DND], DFO, and Indian and Northern Affairs Canada, Northern Affairs Organization [INAC NAO]). Except for the case study of an INAC NAO site, representatives from that department did not participate in the evaluation because INAC NAO has recently conducted its own evaluation of their involvement in FCSAP.
The evaluation considered all FCSAP sites and projects that have been identified, assessed, and/or worked on during the two fiscal years of 2005–2006 and 2006–2007. Some analysis was carried out of sites that received funding under the Federal Contaminated Sites Accelerated Action Plan (FCSAAP) in 2003–2004 and 2004–2005 and continue to receive FCSAP funding today. A number of these sites, such as the Faro and Giant Mines, represent approximately one-third of total federal liability.
The findings of this evaluation are based on input from individuals who are actively involved with FCSAP. The evaluation employed multiple lines of evidence, including a document review; a review of administrative databases; an online consultation with environmental engineering and consulting firms that assess and remediate federal contaminated sites; case studies of seven FCSAP projects; five in-depth case studies of certain issues that arose during the evaluation and merited further investigation; and an interview programme in which approximately 75 key informants from within and outside government provided their views concerning the design, delivery, and success of FCSAP.2 Finally, a panel of three independent contaminated sites experts from the academic and environmental engineering sectors reviewed the evaluation findings.
Overall, the evaluation found that significant changes to the way that FCSAP is designed and delivered are not necessary for the program to meet its objectives. The program was designed in a manner that is appropriate for an interdepartmental program and the objectives it wishes to achieve. In general, FCSAP is well managed, adequately funded, supports Custodians effectively, and has made progress in addressing federal contaminated sites since being launched in 2005.
As the FCSAP Secretariat will begin planning over the next year for the post-2010 phase of FCSAP, the evaluation recommends a number of adjustments to the program be considered to ensure that FCSAP operates as effectively as possible. These recommendations are introduced and discussed in the following pages.
The evaluation found that the FCSAP Secretariat and the Contaminated Sites Management Working Group (CSMWG) have, to a large extent, determined the direction of the program to date. This is in large part due to two factors: attendance by the Assistant Deputy Minister (ADM) at Federal Contaminated Sites (FCS) Steering Committee meetings was lower than anticipated; and a Third Party Expert Panel, intended to provide FCSAP with an external perspective and independent advice, was never implemented. The program does not appear to have suffered, however, since the Directors General (DGs) and Directors who attend meetings in place of ADMs are knowledgeable about contaminated sites, and there have not been major strategic issues suitable for an expert panel to tackle. As the second phase of FCSAP approaches, now is the time for management to think strategically about the future of FCSAP. A few program participants suggested that a DG level committee be struck to provide DGs with a venue to raise and discuss strategic issues (the FCS Steering Committee meetings, as they are currently designed and implemented, reportedly do not provide such an opportunity). Evaluators are of the opinion, however, that creating an additional layer of management may not be the optimum way to proceed. Correspondingly, the evaluation makes the following recommendations regarding the FCS Steering Committee and the use of external experts.
Recommendation 1: The FCSAP Secretariat, in cooperation with the Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider adjusting the FCS Steering Committee from an ADM level to a DG level committee. The discussion of strategic issues should be a regular agenda item for Steering Committee meetings.
Recommendation 2: The FCSAP Secretariat should consider seeking the advice of external experts when developing strategies for the post-2010 period of the program.
FCSAP funded more than $162 million worth of projects in 2006–2007. It is essential that program resources are directed toward the highest priorities and utilized as effectively as possible. The evaluation found that, in general, this is occurring, but room for improvement exists. The evaluation found that Custodians have tended to err on the side of caution and remediate sites when more cost-effective risk management approaches could have been used. To help Custodians address challenging decisions, the FCSAP Secretariat has solicited the assistance of a consultant in developing a Decision-Making Framework, due for release in March 2009. In addition, the current approach to managing sites with large liabilities, or those that have used engineered approaches such as landfills or dams, may leave Custodians exposed to unnecessary risk.
Recommendation 3: For very large projects, the FCSAP Secretariat should consider requiring peer review of remediation/risk management plans and progress. External experts or interdepartmental representatives could be drawn upon to undertake the peer review. Large projects could be considered those in excess of a certain dollar level of liability or ranking score.
Recommendation 4: The FCSAP Secretariat, in cooperation with the Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider making long term periodic monitoring mandatory for the most contaminated FCSAP sites.
FCSAP can also look to the provinces for some practices that help to reduce risk. The provinces of British Columbia, Ontario and Québec require the use of project closure reports for provincial sites, which provide useful summaries of the original problem, the solution pursued, and any remaining risk or concern that should be noted. Some provinces also place a restrictive covenant on land title, identifying the risk management requirements and/or areas. This ensures that any new property owner is aware of the contaminated area and his/her responsibilities for risk managing and monitoring. Without such a covenant, it is possible that a risk managed area or requirement could be overlooked in future years as federal Custodians have a considerable number of pending retirements and associated contaminated sites knowledge may be lost. Although record keeping is in place for the program, evaluators do not consider record keeping to be sufficient to avoid this risk.
Recommendation 5: The FCSAP Secretariat should consider making project closure reports mandatory for all FCSAP projects.
Recommendation 6: The Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider making restrictive covenants on land titles mandatory for FCSAP sites that are being risk managed or monitored.
The evaluation found that the FCSI and IDEA databases capture sufficient data to measure the progress of the program. There are two pieces of information not captured in FCSI that would assist reporting: a first field would identify a site as an FCSAP funded site, and the second field would identify a site as requiring no further action.
Recommendation 7: The Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider adding the capability for FCSI to identify a site as an FCSAP funded site, and to identify a site requiring no further action.
Reporting is currently FCSAP’s weakest point and it threatens to overshadow the hard work and progress the program has made to date. A lack of timely and accurate reporting by some Custodians is making it difficult for the FCSAP Secretariat to evaluate, understand, and communicate the achievements of the FCSAP to date. Despite the efforts of the FCSAP Secretariat to provide guidelines and training on reporting, there are errors in the FCSI data of these Custodians that have not been corrected. A few Custodians have reporting errors that date as far back as the 2005–2006 fiscal year and/or have liability data that differ from that reported in the Public Accounts. In an era when the federal government is placing increasing importance on accountability and results, it is important to ensure that Custodians’ contaminated sites data in the FCSI is complete and up-to-date.
Recommendation 8: The FCSAP Secretariat, in cooperation with the Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider various processes that could be implemented to ensure that Custodians improve the quality and timeliness of both IDEA and FCSI data reporting.
Estimating site liability is a difficult task and there remains some degree of uncertainty among Custodians as to the proper practice. Liability experts consulted for the evaluation pointed out that there are other approaches that may reduce the variability and uncertainty associated with estimating contaminated site liability.
Recommendation 9: The FCSAP Secretariat, in cooperation with the Treasury Board Secretariat, should consider providing additional guidance, training and costing model/templates to help Custodians provide more accurate and consistent liability estimates for projects.
Despite reporting and liability estimation challenges, the picture of federal contaminated sites liability is becoming clearer every year. Based on an analysis of FCSI data, remediation activities at FCSAP sites have reduced federal liability by $361.9 million to date. However, upward liability revisions to existing sites that have been re-assessed, and liability estimates from sites recently assessed for the first time, offset the above mentioned liability reductions. The net result has been a $69.4 million reduction in federal liability associated with federal contaminated sites from March 2006 to March 2007.
The evaluation found that the objective of effectively eliminating federal liability associated with known federal sites by 2020 is probably not realistic for several reasons. Departments, agencies, and crown corporations that participate in FCSAP have other priorities that limit the resources they can devote to addressing federal contaminated sites. There is also a limited pool of qualified consultants that can assess or remediate sites. Many sites are remote and moving people and equipment to them is difficult. It may be several more years before the total federal liability begins to decline, as the liabilities of many sites were underestimated, and other sites have yet to be fully assessed. As planning for the next phase of FCSAP takes place, it would be helpful for the FCSAP Secretariat to have a better understanding of the resource capabilities of Custodians and how many years it will take them to address their sites.
Recommendation 10: The Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider requesting that Custodians articulate in their contaminated sites management plans estimates of the number of sites they have remaining to assess, how many sites they anticipate assessing each year until all sites have been assessed, and their best estimates for when no further action will be required for these sites.
Recommendation 11: The FCSAP Secretariat, in cooperation with the Treasury Board Secretariat, Real Property and Materiel Policy Division, should consider undertaking a comprehensive horizontal program capacity/resource assessment; where gaps emerge, determine risks and develop strategies to address them.
Management Response
The following management response was developed to address the evaluation recommendations.
Recommendation 1:
The FCSAP Secretariat and the Treasury Board Secretariat, Real Property and Materiel Policy Division, agree in principle with the recommendation. Given that the existing ADM-level committee is a Cabinet- and Treasury Board-approved governance structure, changes to its mandate cannot be accomplished under the current program. A DG-level committee has been created to provide additional management support during the development and implementation of the renewed program. Proposed changes to the governance structure will be considered as part of the post-2010 FCSAP program renewal, including a continuing role for the DG-level committee.
Recommendation 2
The FCSAP Secretariat and the Treasury Board Secretariat, Real Property and Materiel Policy Division, agree with the recommendation. As part of the FCSAP program renewal, the FCSAP Secretariat will establish an interdepartmental committee representative of key stakeholders in early 2009 to identify key program issues for the post-2010 period. External experts will then be consulted on significant strategic issues.
Recommendation 3
The FCSAP Secretariat and the Treasury Board Secretariat, Real Property and Materiel Policy Division, agree with the recommendation. In FY 2009-10, the FCSAP Secretariat will undertake a review and gap analysis of existing peer review practices for large FCSAP projects, defined as projects being 100% funded by the FCSAP or having total project expenditures of over $10 million. Requirements for additional internal or external peer review of large FCSAP projects will be identified and considered as part of FCSAP program renewal.
Recommendation 4
The FCSAP Secretariat and the Treasury Board Secretariat, Real Property and Materiel Policy Division, agree with the intent of the recommendation and support long-term periodic monitoring for the most contaminated FCSAP sites. However, the ability to make such monitoring mandatory is constrained by the Federal Real Property and Federal Immovables Act which gives Ministers responsibility for the administration of their department’s real property:
As a result of this Act, departments have an enduring responsibility for the management of their contaminated sites. The FCSAP program, on the other hand, has a fixed time horizon, so the inclusion of additional mandatory requirements would not be sustainable beyond the life of the program. It is, therefore, not within the authority of the FCSAP Secretariat nor the Treasury Board Secretariat, Real Property and Materiel Policy Division, to make long-term monitoring mandatory for any federal contaminated site.
However, the FCSAP Secretariat is currently developing a decision-making framework that will, among other things, assist custodians in evaluating the possible need for long-term monitoring on federal contaminated sites receiving funding under FCSAP. This decision-making framework is expected to be approved by the FCS Steering Committee in 2009-10. The Treasury Board Secretariat, Real Property and Materiel Policy Division, in FY 2009-10 will also investigate the feasibility of tracking long-term monitoring on federal contaminated sites using the Federal Contaminated Sites Inventory (FCSI).
Recommendation 5
The FCSAP Secretariat agrees with the recommendation. In 2009, the FCSAP Secretariat will ask that PWGSC Expert Support develop an appropriate FCSAP closure report template and guidance document for roll-out in the post-2010 period of FCSAP.
Recommendation 6
The Treasury Board Secretariat, Real Property and Materiel Policy Division, agrees in principle with the recommendation. The Treasury Board Secretariat, Real Property and Materiel Policy Division, recognizes the merit of introducing such a concept to ensure that future landowners are well informed of the environmental condition of federal real property as well as any monitoring or risk management requirements that may impact future land use.
The Treasury Board’s Directive on the Sale or Transfer of Surplus Real Property already requires that, prior to seeking an indication of interest in a surplus property, custodians provide interested parties with sufficient information, including the property's environmental and physical condition (section 6.4). It is necessary to examine the feasibility of the application of restrictive covenants to federal real property. The Treasury Board Secretariat, Real Property and Materiel Policy Division, will undertake a more thorough examination of the opportunities and constraints of implementing such a mechanism in consultation with the Department of Justice Canada in the 2009−2010 fiscal year.
Recommendation 7
The Treasury Board Secretariat, Real Property and Materiel Policy Division, agrees with the recommendation. The Treasury Board Secretariat, Real Property and Materiel Policy Division, has established an interdepartmental working group to explore issues relating to the future improvement of the FCSI. The two issues identified in the recommendation have also been highlighted by the working group and solutions will be implemented in 2009.
Recommendation 8
The FCSAP Secretariat and the Treasury Board Secretariat, Real Property and Materiel Policy Division, agree with the recommendation. Accurate and timely information supports the measurement and communication of the achievements of the FCSAP program.
Steps have already been undertaken to identify and address data quality issues in IDEA and the FCSI. Some of these include enhanced training for reporting in IDEA, timely calls for FCSAP program reporting in IDEA, the creation of an FCSI interdepartmental working group, and the creation of conditional frozen allotments for future FCSAP funding for custodians with extensive and ongoing data quality issues in the FCSI. The FCSAP Secretariat is also planning to undertake a program data management survey and gap analysis in 2009 to identify areas for improvement in data management between the FCSI and IDEA, such as opportunities for potential system harmonization in order to ease the reporting burden on custodians.
Recommendation 9
The FCSAP Secretariat and the Treasury Board Secretariat agree with the recommendation. The FCSAP Secretariat is planning to carry out a scoping exercise in 2009 to define the issues and risks related to the reporting of environmental liability estimates in support of the FCSAP program. However, given that the policy responsibility for the accounting of environmental liability rests with the Office of the Comptroller General (OCG) at the Treasury Board Secretariat, the FCSAP Secretariat will work closely with the OCG to develop additional guidance, training, and costing models/templates, to ensure that they will be in line with the new Treasury Board Financial Management policy directions and accounting guidance.
Recommendation 10
The Treasury Board Secretariat, Real Property and Materiel Policy Division, agrees with the recommendation. The Treasury Board Secretariat, Real Property and Materiel Policy Division, will undertake a review of the drafting guidance and revise as required for the next call for Contaminated Sites Management Plans (Spring 2009).
Recommendation 11
The FCSAP Secretariat and the Treasury Board Secretariat, Real Property and Materiel Policy Division, partially agree with the recommendation. The scope of this recommendation involves undertaking activities that are well beyond the current roles and responsibilities of the FCSAP Secretariat or the Treasury Board Secretariat, Real Property and Materiel Policy Division.
However, as part of the post-2010 FCSAP program renewal, an assessment of the existing FCSAP program's human resource issues and gaps will be considered, particularly with respect to the future allocation of departmental program management, Expert Support and FCSAP Secretariat resources. Strategies will be examined to respond to any gaps or program-level risks identified through this analysis.
1 Appendix C presents the FCSAP Logic Model (as of 2007), which presents the key activities and outputs, for each of the three components of the FCSAP, namely, the FCSAP Secretariat, Expert Support and Custodians, as well as shared immediate, intermediate and long-term outcomes.
2 Appendix A presents a list of all key informants interviewed as part of this formative evaluation.