This page has been archived on the Web
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
Evaluation of Enhanced Security for the Transboundary Movement of Hazardous Waste: Lessons Learned
July 4, 2008
Report Clearance Steps
Planning phase completed - January 2008
Report sent for management response - March 2008
Management response received - May 2008
Report completed - May 2008
Report approved by Departmental Evaluation Committee (DEC) - July 2008
The Evaluation Project Team, led by Martine Perrault and including John Morris under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights and comments crucial to this streamlined evaluation.
Prepared by the Evaluation Division, Audit and Evaluation Branch
Acronyms used in the report
- Assistant Deputy Minister
- Audit and Evaluation Branch
- Canada Border Services Agency
- Canada Customs and Revenue Agency
- Commission on Environmental Cooperation
- CEPA 1999
- Canadian Environmental Protection Act, 1999
- Chief Financial Officer
- Chief Information Officer
- Chief Information Officer Branch
- Canadian Notice and Manifest Tracking System
- Departmental Evaluation Committee
- Electronic Data Exchange
- Export and Import of Hazardous Waste and Hazardous Recyclable Material Regulations
- Export and Import of Hazardous Wastes Regulations
- Environmental Protection
- Environmental Protection Agency
- Environmental Stewardship Branch
- Hazardous Waste Task Force
- information technology
- Public Safety and Anti-Terrorism
- Scientific Applications Development Division
- Treasury Board
- Treasury Board Secretariat
- Transboundary Movement Branch
- Transboundary Movement Program
- United Nations Centre for Trade Facilitation and Electronic Business
- United States
- United States of America
- Waste Reduction and Management Division
In the 2007-2008 fiscal year, Environment Canada's Audit and Evaluation Branch (AEB) conducted an evaluation of Environment Canada's Public Security and Anti-Terrorism (PSAT)-funded activities pertaining to enhanced security for the transboundary movement of hazardous waste, from their inception in 2001 to the present. This evaluation was part of Environment Canada's Audit and Evaluation Plan for 2007-2008, which was approved by the Departmental Audit and Evaluation Committee on April 18, 2007.
The objectives of this evaluation were to
- meet Treasury Board (TB)'s requirement that all activities for which Environment Canada received PSAT funds undergo an evaluation;
- meet AEB's commitment to evaluate, during the 2007-2008 fiscal year, the Enhanced Security for the Transboundary Movement of Hazardous Waste component of PSAT funding received by Environment Canada; and
- produce useful lessons learned on this project's implementation, tracking and reporting for future implementation of similar types of activities within the Department.
The evaluation was intended to address the issues of relevance, design and delivery, success and value-for-money. However, implementation of a systematic evaluation approach was complicated by the absence of any clearly articulated project logic, expected results and indicators, by the limited performance information and project documentation available, particularly as pertains to financial tracking, and by the loss of some of the project's corporate memory due to manager turnover and organizational transformations at Environment Canada. For these reasons, the evaluation focused on describing how the project was implemented, to provide context for the lack of results information, and on identifying lessons learned.
This evaluation relied on a review of project documents and Internet publications, as well as interviews with key staff, managers, stakeholders and partners involved in the implementation of this project since its inception.
Regulations dictate that transboundary movements of hazardous wastes and hazardous recyclable materials across Canadian borders be permitted and tracked by the appropriate authorities to ensure that they do not pose a security threat or a threat to the environment. In 2001, PSAT funds were given to Environment Canada's Transboundary Movement Branch (TMB) to investigate the feasibility of improving its paper-based permitting and reporting system for hazardous waste shipments with the introduction of a smart card system. The smart card initiative was intended to provide federal authorities with paperless, real-time access to information on the status of the shipments and enable quicker and more effective enforcement of applicable regulations. Funding for the project started in 2001 and was established at an ongoing level of $700,000 annually.
The smart card project confirmed the potential utility of smart cards in tracking hazardous wastes but found that neither the Canada Customs Revenue Agency (CCRA) nor the regulated community were ready to invest the resources required to implement this initiative. As a result, TMB managers decided not to pursue implementation of the smart card technology and agreed on a new direction for continuation of the PSAT project, namely the redesign, modernization and updating of its existing tracking system, and the continued development of common North American data standards and elements.
The program was successful in redesigning its tracking system, the Canadian Notice and Manifest Tracking System (CNMTS), to meet the more stringent controls introduced in 2005 by the new Export and Import of Hazardous Waste and Hazardous Recyclable Material Regulations (EIHWHRMR). In addition, updates were made on an ongoing basis to maintain the current system. Program managers and staff have also progressed, with USA and Mexico counterparts, on the development of common North American data standards that will enable the sharing of hazardous wastes movement data across jurisdictions. Through their lead participation in the Hazardous Waste Task Force (HWTF) within the Commission on Environmental Cooperation (CEC), they plan to be ready to move data across jurisdictions in a pilot study by spring 2009.
All work has stopped, however, on modernizing the CNMTS. In 2005, the contractor hired to modernize the CNMTS to facilitate exchange of data in real-time concluded that it could not complete the agreed-upon deliverable under the original contract value. A legal dispute ensued and all work on this component has been suspended. To date, the program is still relying on paper-based permitting and reporting of hazardous waste and hazardous recyclable material shipments.
A number of external factors have also affected the program's ability to meet its ultimate PSAT project objective of moving to a paperless tracking system. First, TMB was replaced in 2006 by a new unit, the Waste Management Division (WMD), Environmental Stewardship Branch (ESB), which was further restructured in early 2007 into the Waste Reduction and Management Division (WRMD). During this restructuring phase, the responsibility for the PSAT project was split across the WRMD and the newly created Chief Information Officer Branch (CIOB). The WRMD retained leadership of the project while CIOB's Scientific Applications Development Division (SADD) was responsible for providing the technical expertise required to support the project. This transformation resulted in a loss of IT technical competency at WRMD and in its decreased ability to manage a significant IT project. Furthermore, no formal delineation of roles and responsibilities was established, further affecting management's ability to be accountable for project results.
These challenges were further compounded by the fact that, from 2005-2006 onwards, PSAT funds were no longer identified and tracked separately. This came as a result of broad changes that took place at that same time within the Department. Following the departmental changes, coding used until then to identify the budget amount allocated to this project and subsequently used to track project expenditures was no longer used. Representatives from the Finance Branch confirmed that, until recently, no mechanism existed to determine what portions of the departmental A-Base budget had been transferred to the responsible Board for this specific project. There is no evidence or consensus across key informants from ESB, CIOB and Finance Branch, however, to explain why an alternative tracking mechanism was not implemented to track annual budget allocations and expenditures for the project. The conclusion that can be drawn is the need for ESB, CIOB and Finance Branch to work together to resolve this ongoing issue.
Findings from interviews and a review of available documents show that this project addressed a relevant and ongoing need for improving access to real-time data on transboundary movements of hazardous wastes. Although there is no documentation showing how the decisions were made, modifications brought to the project design were justified and consistent with the overall project objectives.
However, the project lacks appropriate reporting, documentation of its initial design and clear articulation of expected deliverables and expected outputs and outcomes. Furthermore, no evidence was found of a formal governance structure for guiding and supporting management and decision making for this project. Also, the absence of a clear delineation of roles and responsibilities and the uncertainty surrounding allocation of PSAT funding undermine managers' ability to be accountable for PSAT project results.
Despite the above limitations, the project is presently moving in an appropriate direction and putting in place the required processes to achieve its ultimate goal. In particular, the TMB/WRMD has refined its understanding of the requirements for successfully developing and implementing a paperless tracking system and the SADD has developed more rigorous methods for implementing and ensuring due diligence in the management of IT projects. Both partners have developed a draft project charter and service-level agreement that should, when signed, clarify their respective roles and responsibilities and expected deliverables. Finally, Environment Canada has established itself as the technical lead on the CEC HWTF collaboration, which has produced the first building blocks of a common North American data standard agreement.
Specific lessons learned around the governance and management of relatively small-scale projects are offered below and are intended to serve as a guide for the implementation of similar projects at Environment Canada in the future:
- The methodology that drove the development of the smart card project was to use an IT solution to address a business concern. In this specific case, too much emphasis was placed on the IT solution, which led to an IT team driving the project and designing a business solution. A different approach would see the recognition of a business concern and its full understanding before an IT solution is accepted. The implementation of this methodology would allow Environment Canada to better understand the nature of its concerns and then clearly define its business requirements before engaging technical experts to craft a solution.
- Given that technical components of IT projects are now the responsibility of CIOB, while content expertise remains with the substantive programs, the need is more prevalent than ever to develop service-level agreements and project charters to clearly articulate respective mandates and budgets, as well as expected project deliverables and outcomes. A more detailed project design document articulating the project's logic, expected immediate, intermediate and long-term results, and performance indicators would have helped managers monitor progress, make decisions and report on results more effectively, especially when the project design was modified.
- Managers are unable to identify what proportions of their budgets are tied to specific project commitments. This limits their ability to allocate appropriate resources, to meet specific reporting and accountability requirements, and to make decisions regarding small ongoing projects such as the PSAT project.
The following recommendations are put forward to guide the resolution of the issues that hinder the progress of this initiative and to ensure its continuation and success.
1. The Environmental Protection (EP) Board needs to confirm departmental commitment to pursue this project's overall objectives. If a commitment is made to continue this project, the Assistant Deputy Minister (ADM), ESB and Chief Information Officer (CIO) need to ensure that WRMD and SADD establish clear project accountability and governance, delineate roles and responsibilities, and identify ongoing resources for the project.
Clear governance, accountability and definition of roles and responsibilities are essential for effective project management, particularly for IT projects involving the participation of two different work teams. For this PSAT project, effective collaboration between WRMD and SADD was hindered by the absence of a formal agreement on their respective roles and a decision-making protocol. Such a formal agreement will establish the basis for effective future collaboration on this project.
2. Once the project governance and accountability have been formalized, the ADM, ESB and CIO need to ensure that WRMD and SADD collaborate to develop a path forward strategy to guide the future continuation of this project. This strategy should include a clearly articulated project logic, identifying activities, expected outputs and outcomes, and key deliverables and milestones.
A clear project design and well defined targets and milestones by which the project's progress can be assessed are essential to effective project planning, monitoring, reporting and accountability. These elements were not sufficiently detailed and articulated for the PSAT project and should have been revised every time the project focus was modified. A path forward strategy containing these elements will provide a common vision and work plan to guide effective management of this project in the future.
3. The Assistant Deputy Minister, ESB, CIO and Chief Financial Officer (CFO) need to work together to strengthen WRMD and SADD's ability to identify and track PSAT funds in the context of the Enhanced Security for the Transboundary Movement of Hazardous Waste project.
The ability to identify and track funding allocated to a specific initiative and tied to specific results commitments is a key feature of good governance, accountability and effective project planning. This evaluation showed that the ability to identify and track PSAT financial resources is currently lacking.
4. The Assistant Deputy Minister, ESB and CIO need to ensure that WRMD and SADD establish procedures for tracking and documenting decision making pertaining to this project and for ensuring systematic safekeeping of all relevant project documents.
Performance reporting and evaluation rely on the availability of complete and accurate documentation on a project's activities, decision making and results. The scope of this evaluation was affected by the limited available documentation on all aspects of this project's design, implementation and results. Proper documentation will assist project managers in producing a credible performance story and meet future project accountability requirements.
Environment Canada, through the programs under the Waste Reduction and Management Division (WRMD) is dedicated to protecting human health and the environment for the high level of risk posed by the mismanagement of hazardous waste and hazardous recyclable material. In addition, Environment Canada recognizes the need for activities which enhance the security of the Canadian public and contribute to anti-terrorism efforts Canada-wide. The hazards associated with wastes and recyclable materials are all the more real and relatively the same as hazardous products which can be used by individuals or organizations in a fashion that would negatively impact Canadian security. Therefore, the programs and activities of the WRMD in monitoring and tracking international movements of hazardous waste and hazardous recyclable material do support the PSAT objects.
The recent program evaluation was designed to consider various activities undertaken by WRMD and Chief Information Officer Branch (CIOB) for the implementation initially of a smart card mechanism and then the modernization of the electronic Canadian Notice and Manifest Tracking System (CNMTS), in an effort to move towards real-time tracking of international movements of hazardous waste and hazardous recyclable material. The evaluation findings provide helpful direction on the transboundary movement program and highlight the need for an integrated governance structure that should lead to a more efficient and cost effective project management.
As such, the Assistant Deputy Minister (ADM) of the Environmental Stewardship Branch (ESB), the Chief Information Officer (CIO), and the Chief Financial Officer (CFO) accept the evaluation and its recommendations and have provided a preliminary implementation plan for the recommendations.
Recommendation 1: In response to Recommendation 1, by June 6, 2008, the WRMD and SADD will establish and present to EP Board, for its approval, an overall strategy to develop a project workplan that will include:
- Articulation of the Enhanced Security for the Transboundary Movement of Hazardous Waste project objectives for EP Board's approval, in light of the developments since the inception of the project in 2001, and as part of Environment Canada's assessment of its priorities in the context of the Department's Strategic Review
- Deliverables for 2008-09 (initial phase of the project), which will include a detailed path forward to develop an overall strategy for presentation to EP Board by October 31, 2008
- Roles and responsibilities for both the WRMD and SADD with respect to the deliverables of the initial phase of the project
- Governance structure for decision making with respect to the initial phase of the project
- Resource requirements to develop the project workplan for October 31, 2008
Recommendation 2: Following the decisions arising out of the EP Board presentation on June 6, 2008 and assuming approval of the overall workplan, the ADM, ESB and CIO will develop a detailed path forward strategy covering a period of 3 years and present it to EP Board for its approval by October 31, 2008. This strategy will build on the overall project workplan approved by EP Board on June 6, 2008 and will clearly identify activities, deliverables and milestones, resource requirements, and roles and responsibilities for meeting expected outputs and outcomes of the project.
Furthermore, the strategy will take into consideration future and concurrent initiatives associated to this project, including but not limited to:
- Common TBS and Environment Canada software and electronic information standards
- Government initiatives such as the initiative to reduce paper burden
- Future regulatory improvements to the Export and Import of Hazardous Waste and Hazardous Recyclable Material Regulations
- Departmental initiatives such as the Environment Canada Single Window reporting initiative
- Component development for data exchange with international partners (CEC Hazardous Waste Task Force)
- Component development for data exchange with other government departments (CBSA Single Window)
- A report on progress will be prepared annually, as required by TBS. The path forward strategy will also be revised as needed to reflect the evolution of the project
Recommendation 3: Following the decisions arising out of the EP Board presentation on June 6, 2008 and assuming approval of the overall workplan, the ADM, ESB, CIO and CFO will determine by June 6, 2008 how the mechanisms currently available within the departmental financial systems can be best used by WRMD and SADD to ensure that PSAT funds can be:
- Clearly identified within the budget of the two Divisions; and
- Tracked when they are used in activities that support the delivery of the PSAT project within both WRMD and SADD.
Subsequent to implementation the appropriate directors general within ESB, CIOB and CIO will on a regular basis review these activities and resource allocations, to ensure that the mechanisms being used continue to meet Managers' needs.
Recommendation 4: Following the decisions arising out of the EP Board presentation on June 6, 2008 and assuming approval of the overall workplan, the ADM, ESB, CIO and CFO will establish, by October 31, 2008, a document management system to ensure that the project follows proper best practices including quality assurance activities in accordance to departmental policies and/or guidelines that will:
- Identify roles and responsibilities with respect to the management of documentation
- Ensure the accuracy and validity of the documentation between WRMD and SADD with respect to the Enhanced Security for the Transboundary Movement of Hazardous Waste project
- Ensure the safe keeping of documents (electronic and/or paper) for all parties relating to the project
- Monitor project activities at set intervals and/or milestones of the project
- Establish a shared drive on Environment Canada servers for the specific use of information sharing on matters with respect to the continued development of the project
- Date modified: