Environment Canada's (EC) Audit and Evaluation Branch completed the evaluation of the Georgia Basin Action Plan (GBAP) in April 2007. This project was selected for evaluation by the Departmental Audit and Evaluation Committee (DAEC) of Environment Canada in November 2005. The DAEC directed that an approved evaluation framework be applied to the GBAP, one of the programs under Priority Ecosystems.
The Georgia Basin Action Plan, which evolved from the earlier Georgia Basin Ecosystem Initiative, was implemented on April 1, 2003, to establish priorities and undertake initiatives that positively influence the state of the natural environment, economic growth and social capital within the Georgia Basin and Puget Sound region. The GBAP is the second five-year phase (2003-2008) of collaborative programming for the Georgia Basin.
There were four broad issue areas of focus for the evaluation, namely:
Program relevance — the degree of alignment with the directions and priorities of Environment Canada, the clarity of the intent of the GBAP, as well as the continuing rationale for the program given prevailing areas of need and the potential for overlap with other initiatives.
Success — perceived results of the GBAP, both intended and unintended, in areas supporting the mandate of the program.
Cost-effectiveness — the potential for the GBAP to use resources more effectively in the pursuit of its mandate.
Design and delivery — the clarity of activity, accountabilities, expected deliverables and intended results of the GBAP. In addition, process considerations pertaining to the allocation of resources, management of risk, monitoring and reporting, and the leveraging of partnerships were considered.
This evaluation of the GBAP is summative in nature. It rigorously applies the 30 evaluation questions from the evaluation framework and examines all four evaluation issues noted above.
To test and support the approved framework, the evaluation employed the following data collection methods:
document reviews in order to assess whether the evolution of the GBAP, along with management issues and information on the program and its operations, has been clearly understood and accurately reported;
telephone and in-person interviews using structured guides with internal Environment Canada staff and external stakeholders and partners;
one facilitated focus group; and
a survey of Project Team Leads.
Below is an overview of the findings in a summary format according to the four main issues — relevance, success, cost-effectiveness and design and delivery.
The fundamental design elements which translate the principles of an ecosystem approach into a well-defined program structure are lacking (not adequately defined in documents both for Priority Ecosystems [PE] and for the Georgia Basin Action Plan [GBAP]).
There is evidence showing a role for government in this program area.
A valid role exists for the federal government, based on documents and interviews.
The public interest is served by the program; however, there is no apparent comprehensive and analytical examination of targeting reach, even though partnership constitutes one of the program principles. The unbundling process obfuscates the clarity and roles of a program such as the GBAP, an area-specific ecosystem initiative.
Theoretically there is a link associating this program with departmental outcomes from the OPP to the OPG, as well as Board outcomes and Board priorities.
Both documentation and staff interviews illustrate that there is a connection between this program and overall departmental strategic outcomes.
There is wide-ranging opinion on whether adjustments to the program are necessary to ensure better alignment with departmental priorities.
Due to a lack of relevant documentation, we cannot comment extensively on whether all OPPs within the scope of the OPG need to exist.
To answer the question of program duplication, EC staff universally does state that no duplication exists. This evaluation did not specifically engage other methodologies to consider program duplication.
Given that documentation of outcomes and project deliverables was not readily available, comprehensive or complete, little to no objective evidence of outcomes exists. We find some success in achieving outcomes at a project level, anecdotally.
Neither rigorous nor systematic performance measurement data were supplied by the program; therefore, it is difficult to comment on attribution of program outcomes to outputs.
There is a potential implication for Canada's economic growth and competitiveness associated with the programs; however, this linkage is at the early stages and no definitive attribution/conclusions can be drawn. The GBAP's environmental results could not be readily demonstrated, making the further links to economic performance that much more tenuous.
Broad and wide-ranging unanticipated outcomes have been identified as occurring from the program based on interview comments.
Alternative delivery approaches were not specifically researched as part of the evaluation methodology and documentation on this does not exist.
Given that efficiency of a program is a calculation based on the amount of outputs generated by dollar of resources input, and we could not make such a calculation, the question of whether GBAP is efficient cannot be answered. This is problematic. It was found that there was no consistent or complete available listing of projects (and their outputs) for the GBAP. It was also found that the resources available were increasingly under-spent from FY2003-04 to FY2005-06. In the absence of performance reporting, success/effectiveness cannot be demonstrated. This brings cost-effectiveness/value for money into question.
Cost recovery is not an issue for this evaluation given that no specific clients receive custom benefits.
Given that no attribution between program activities/outputs and outcomes can be found, and in the absence of performance reporting, success/effectiveness cannot be demonstrated. Thus cost-effectiveness / value for money is brought into question, and the structure and ecosystem approach for PEs remain undocumented. All this leads one to conclude that there is no demonstrable evidence that Canadians are getting value for money.
It is difficult to state whether the program is affordable.
Certainly given the requisite departmental OPP/OPG structure, there are documented deliverables and results for the program. However, basic information such as an inventory of projects and their outputs as well as PTLs is not readily available. Communication of such information to PTLs is also an issue.
Some important gaps in the logic of the program exist (e.g., development/application of Priority Ecosystem Approach). These gaps cloud plausible attribution of outcomes.
There is no apparent systematic decision-making approach to project selection and resource allocation.
There is documentation of a risk management strategy but the extent of actual management actions on this is questionable. Moreover, there is no formal vision to date, partners are elusive and the unbundling is weakening connectivity to projects.
There is no apparent documented analysis of financial capacity requirements. There is no apparent need for additional financial capacity (in 2005-2006, actual vs. budgeted left a 22 percent surplus of funds). No data exist on human resources capacity analysis, although there seems to be a significant level of staff turnover in the GBAP office.
Theoretically, there appears to be a linkage between the program and the Department's strategic positioning/former CESF pillars.
In terms of partnerships, which is one of the principles of the program, an extensive survey of GBAP partners was planned for some 50 partners of the program; however, the survey was cancelled as only 13 partners could be successfully located. The evaluation was initially planned to have 5 focus groups with a total of 50 participants, composed of co-deliverers familiar with the GBAP; however, only one focus group with 10 participants could be assembled. No industry partners were identified by EC. Given that it was difficult to locate partners, there is an apparent issue with regards to reach for the program. This is especially problematic given that the program is based on partnerships.
There is no apparent documentation of a complete program design upon which to base an assessment of the actual delivery and thus comment on the consistency of program design versus delivery.
In terms of EC management and staff being supportive of the program, there are wide ranging opinions among staff on this issue. Surprisingly, there were problems finding the actual PTLs for GBAP projects; only 19 were successfully surveyed.
There is a management structure defined and operating. However, there is no extensive systematic decision-making approach/structure to ensure active links of the GBAP to management that would provide responsive support to the program.
The above series of findings leads to the following conclusions:
It is appropriate that the federal government be involved with these types of activities which preserve and protect the environment of Canada, as these activities both serve the public interest and respect the overall division of powers across governments.
However, the PE as well as the GBAP lack fundamental design elements which a program is expected to have, such as a vision and defined methodology for executing a Priority Ecosystem Approach (PEA), as well as criteria and systematic methods for the selection of activities. In addition, the elements of planning, implementing, monitoring, and reporting are not well carried out.
Partnerships constitute a principle for both the Priority Ecosystems and the Georgia Basin Action Plan; yet such extensive partnerships could not be located for the evaluation. The difficulty in securing this information indicates a problem for the program.
The financial basis of operations of the GBAP governance generates fundamental questions about how resources are used and linked to results. The activity costs some $5.5 million annually. Some 78 percent of these costs are classed for salary and, operations and maintenance with no further cost details readily available. Funds have not been spent as allocated, with some 10 percent, 7 percent and 20 percent of such funds identified as surplus at year end for the last three years.
The GBAP lacks a rigorous and systematic performance measurement of its operations, making analysis and plausible attribution to outcomes difficult.
Two significant and global recommendations are made:
Given that the Department has committed to an ecosystem approach, the lead of the Priority Ecosystems (PE) OPG along with the leads of the other ecosystem OPPs including GBAP should:
Establish clarity on strategic planning and structural issues of the ecosystem approach. This should include clear articulation of the roles and responsibilities across OPGs, OPPs and unbundled OPPs, direction setting, methodology, outcomes, results structure and communications. The results should be reported to the Ecosystem Sustainability Board for approval by January 2008.
Undertake a management review of all ecosystem initiatives to ensure appropriate performance measurement, reporting and associated accountability. This should be reported to the Ecosystem Sustainability Board for approval by December 2007
The PE OPG and, specifically, the GBAP OPP need to seek greater precision on the vision as well as a tightened operational definition of what specific results are being sought by using an ecosystem approach, and put the vision and expected results into an operational program. This needs to be completed prior to consideration of the initiation of any new program development.
As well, the unbundling exercise has resulted in many projects no longer being reported as part of the GBAP OPP. The results of those now unconnected projects still do need to be monitored for results delivery. The need for the current structure and role of regional ecosystem initiatives as well as their functions with respect to the coordination of unbundled ecosystem initiatives' activities/results therefore needs to be closely re-examined. Given that the Department has a complete results-based management system, the unbundling exercise leaves only the governance issue under the PE OPG. The governance role with respect to the individual ecosystem OPPs and the OPG needs to be examined for possible streamlining and efficiencies.
The OPG should examine these results for the GBAP initiative in light of the other five ecosystem initiatives under its direction. This will be important given the development of a National Ecosystem Framework in EC (ES Board Deck March 2007) and fundamental to the design of departmental RBM programs. There is active discussion of an interim two-year extension of the current ecosystem initiatives followed by a renewal process in 2009-2010. However, the conclusions and recommendations of this report need close scrutiny and reflection in the context of that approach.
Over a year ago, the Environmental Sustainability (ES) Board called for a new EC ecosystem approach and a Priority Ecosystem Initiative Management Framework.
Under the leadership of the PE OPG, an Ecosystem Approach as a management model to environmental management was articulated in 2006 and presented to both ES Board (January 2007) and the Weather and Environmental Services (WES) Board (May 2007). The implementation of this approach is starting and will be done on a continuous basis using EC's management structure and planning process.
The PE OPG is also working, with participation from other OPGs, on a Priority Ecosystem Initiative Management Framework. The purpose of the framework is to:
encourage the systematic, repeatable and defensible (1) selection of Priority Ecosystems and Priority Ecosystem Initiatives (PEIs) and (2) determination of EC's level and type of involvement (roles and responsibilities) from a range of possible levels; and
present direction and basic guidelines for PEI planning, design, implementation, management and reporting in a nationally consistent manner, consistent with the Ecosystem Approach.
It is our intent that the framework, once developed, be applied to existing Ecosystem Initiatives, and to potential new ones, by March 2008. This should lead to a more common approach to selection, design and delivery of PEIs in the Department.
As they are progressively implemented, we are confident that these two initiatives should address most of the observed weaknesses related to recommendation 1(a): roles, responsibilities, direction setting, methodology, etc.
With regard to recommendation 1(b) a process has already been launched with the assistance of the Evaluation Division of the Audit and Evaluation Branch (A&E), and will be implemented to provide assurance to ES Board members that the issues noted in the evaluation with respect to the GBAP (presence of a management framework, appropriate information on performance, partners and projects) are being considered and addressed to the extent that they apply to other PEIs. This report will be provided to the ES Board by the end of October 2007.
Once available, the OPP leads will also be actively involved in the application of the PEI Management Framework to the existing PEIs (March 2008) and we strongly believe that this will have a very positive effect towards strengthening, overall, the governance and effectiveness of PEIs.
The lead of the GBAP OPP, with support and coordination from the lead of the Priority Ecosystems OPG, should undertake actions to urgently deal with improvements to the planning, measuring, and reporting systems outlined below by December 2007 and report those to the Ecosystem Sustainability Board for approval.
The entire spectrum of planning, measuring, and reporting of results of the GBAP OPP needs to be rigorously documented and managed during the remaining timeframe of the GBAP (April 2008) whereby close tracking of all projects, results, and deliverables becomes readily available and actively used in program decision-making. A comprehensive listing of results achieved at the conclusion of GBAP is required so as to allow for a post-mortem assessment of value for money invested.
Given that the population being targeted by a program is crucial to achieving successful outcomes and that this was problematic for the GBAP, the OPP should do an in-depth analysis of reach and report on whether its planned versus actual reach was well-aligned and achieved during program delivery. Such an examination of reach by GBAP may also benefit other ecosystem OPPs as well as the overall OPG.
As a first step in response to the evaluation finding, the ES Board requested that all spending on the GBAP initiative except for select items or elements be suspended. This fall, the results of an analysis carried out by the OPG Leads and the Regional Director General, Pacific and Yukon Region on the planned path forward for the GBAP would be discussed with the Board.
The Georgia Basin Coordination Office (GBCO) has been tasked with establishing and implementing an action plan to urgently deal with improvements to the measuring and reporting of results and outcomes from the GBAP. A preliminary report will be tabled to the ES Board in December 2007, followed by a full report by April 2008. This same exercise will provide the material for a report that documents the outputs and outcomes of GBAP projects, and provides a final report on the 5-year program. This report is targeted for completion by April 2008 in order to provide meaningful summative reporting on the program's achievements.
The GBCO is also operationalizing the performance measurement framework developed in 2004, which provides measures related to outcomes identified in the GBAP logic model.
GBCO staff will undertake an analysis of reach, as they go forward, to document program activities, status, outputs and outcomes.
This analysis will be based on the Communication and Outreach Strategy for the GBAP that was developed through a concerted effort in 2004 and articulates the planned reach. This information will be collected and reported-on in the five-year wrap-up report.