Environment Canada’s Evaluation Division, Audit and Evaluation Branch, conducted an evaluation of the EcoAction Community Funding Program. The evaluation was identified as part of the Departmental Audit and Evaluation Plan approved in April 2008 and was conducted as a requirement for program renewal, focusing on the fiscal years from 2004–05 to 2008–09. The EcoAction program will be seeking renewal for its terms and conditions in the spring of 2010-11.
EcoAction focuses on supporting community-level projects led by non-profit organizations that are expected to have measurable, positive results for the environment. While national in scope, the program is managed in the regions through a network of Environment Canada offices across Canada. The objective is to offer project support to EcoAction clients throughout the application process and to monitor funded projects closely.
Eligible projects may be funded up to a maximum of $100,000 per project which, by design, makes EcoAction a funder of small projects (the average funding is about $25,000 per project). In order to be funded, applicants must lever at least 50% of the total value of the project from sources other than the federal government. The maximum duration for a funded project is two years and projects that are designed to become self-sustaining after this period are encouraged.
The objectives of the EcoAction program are to:
Funded projects must have positive intended results for the environment in one of four priority areas of the EcoAction program: Clean Air, Climate Change, Clean Water, and Nature.
The Evaluation of the EcoAction Community Funding Program assessed the relevance, success and cost effectiveness, and to a lesser extent revisited the design and delivery, of the EcoAction program. The evaluation was designed to determine whether the program:
Data were collected for the evaluation using multiple lines of evidence. These included a document/file review, secondary data analysis of information recorded in the Management Information System (MIS), analysis of the 2008 client survey (n = 126), 18 key informant interviews with staff and stakeholders, and a survey of non-funded applicants (n = 154). Notwithstanding the strengths related to using multiple lines of evidence, some challenges and limitations were encountered, including difficulties in attributing findings to the EcoAction program and inconsistencies in MIS data.
Overall, evaluation findings indicate that EcoAction provides an important source of support for community-based environmental projects, as described below. However, the out-of-date program data in the MIS and the limited data available on each of the program’s indicators make it difficult to show program results and determine the achievement of outcomes.
Evaluation findings are summarized in the following sections by evaluation issue.
a) Relevance
EcoAction supports community-level, environmental projects that reflect Environment Canada priorities, such as the reduction of greenhouse gas (GHG) emissions, improvements in air and water quality and protecting the species and their habitat. Those involved in the program or projects report that without EcoAction, a variety of projects would not exist. Some of the evaluation’s key findings include:
Role of Federal Government: Federal participation in environmental protection and restoration is aligned with federal priorities. The federal government’s environmental agenda and the delivery of the EcoAction program are led by Environment Canada, which is the appropriate department for this role given its mandate to preserve and enhance the quality of the natural environment.
Continued Need for the Program: The EcoAction program and the funding it provided was critical for both those who received funding and those who did not. According to survey results, 86% of funding recipients report their projects would not have been carried out or would have been significantly reduced in size or scope without EcoAction. For projects that were not funded, 70% of applicants indicate that their projects were not carried out or were significantly reduced in size or scope without EcoAction. These findings show that the program addresses a financial need for these community-based groups. This need is also reflected in the high number of applications that are received relative to the number of projects that are eventually funded: EcoAction receives about 400 applications for funding each year, of which roughly 40% are approved.
b) Success
Performance information on projects funded through EcoAction should provide a clear indication of whether or not the intended outcomes of the program have been achieved. However, analyses for this evaluation were limited by the quality of the data available. These data are largely based on the opinions of program staff and EcoAction clients, often not quantified in terms of environmental outcomes, and not validated. The following are key findings:
c) Cost Effectiveness
The cost effectiveness of the EcoAction program was difficult to assess because accurate program expenditures (both operating and salary) are difficult to determine and because of limitations in measuring achieved outcomes. As a result, cost effectiveness was examined indirectly by assessing the efficiency of the EcoAction program. Findings indicate that, while the program’s design necessitates high administrative costs, some improvements may be made to improve the cost efficiency of the EcoAction program.
Value for Federal Dollars Spent: Projects funded through EcoAction provided value for money by leveraging $2.26 on average from community partners for every dollar contributed by the program, exceeding the program’s minimum requirement that projects obtain at least 50% of their funding from other sources. Evaluation findings, however, indicate that funds spent on salaries and Operations and Management (O&M) represent $0.39 of every contribution dollar spent on the program. While this ratio is closely aligned with the design of the EcoAction program, it is higher than estimates for other Grants and Contributions (G&C) programs, such as Environment Canada’s Invasive Alien Species Partnership Program (IAS) which spends $0.15 and the Habitat Stewardship Program (HSP) which spends $0.13 on administrative costs for every dollar in contribution funds. EcoAction is designed to have a large number of Full-time Equivalents (FTEs) to accommodate its heavy emphasis on service. For example, program staff are continuously engaged with funded community groups throughout each phase of the project to administer the ongoing monitoring and reporting that is required. While part of the program design, this emphasis on client service leads to a large cost per application as well as per funded project.
Avoiding Duplication: While they were aware of similar funding programs, interviewees felt that EcoAction did not duplicate these programs. Program staff report that, unlike other programs, EcoAction’s general priorities and flexibility allow for funding a wider range of projects compared to other, more targeted funding programs with more restrictive eligibility criteria. In this way, EcoAction provides funding opportunities that otherwise may not be available for some community groups. By implementing the requirement for leveraged funding from partners, EcoAction provides a complementary source of financial and in-kind support for community organizations that may require additional resources to carry out their projects. For example, EcoAction covers a variety of expenses like human resource costs that may not be allowed according to the funding criteria of the partner organization.
Improvements to Cost Efficiency: Overall, interviewees felt that EcoAction funds are being used efficiently and effectively as a result of reporting requirements and project evaluation processes currently in place to ensure accountability. However, suggestions were provided for improving the cost efficiency of the program, including streamlining processes for approving applications and considering ways to lower administrative costs.
d) Design and Delivery
It should be noted that this is an impact evaluation, conducted as a requirement for program renewal and was limited in scope to explore design or delivery issues.
Client Satisfaction with the Program: EcoAction clients were generally satisfied with the program and services received. More specifically:
Just under half (49%) of funded applicants agreed while 20% disagreed that their application review was timely. Non-funded applicants reported similar responses. These findings point to the existence of delays in the review process for EcoAction applications. Delays in the departmental financial approval process were also identified by program staff and other interviewees as a barrier impeding the ability of funded projects to achieve their objectives;
Overall, a higher percentage of funded applicants found program staff to be helpful or very helpful during the application process compared to non-funded applicants. For example, 83% of funded applicants report that program staff were helpful or very helpful in determining the eligibility criteria of their projects while 59% of non-funded applicants found them helpful. Similarly, 76% of funded applicants reported that program staff were helpful in completing the application properly compared to 53% of non-funded applicants;
Findings indicate that both funded and non-funded applicants find some of the application forms and tools difficult to understand. A higher percentage of funded applicants found both the eligibility criteria (67%) and the application guide (53%) easy to understand compared to non-funded applicants (46% for both eligibility criteria and application guide). However, less than 45% of both funded and non-funded applicants report that the application forms were easy to complete and that the EcoAction website was easy to navigate.
Three recommendations were developed for the Ecosystem Sustainability (ES) Board based on the evaluation findings and conclusions.
RECOMMENDATION #1: It is recommended that program forms and tools be improved to make them easier to understand and to use.
Evaluation findings show that, while clients were satisfied with the services received by EcoAction staff, they were less satisfied with the ease of understanding of program forms and tools. In particular, funding applicants noted difficulties navigating the EcoAction website, understanding the application guide and eligibility criteria as well as completing the application form. Funded applicants also identified difficulties completing reporting forms used to monitor their projects.
RECOMMENDATION #2: It is recommended that roles, responsibilities and processes be examined to identify opportunities for clarification and increased efficiencies.
The EcoAction program was designed to fund projects at the community level with an emphasis on client service and this contributes to its high administrative costs to operate the program. The identification of best practices, particularly in the regions, and areas where streamlining the delivery process might be possible could help to improve the overall efficiency of the program. As well, the evaluation identified some uncertainty among program staff around roles and responsibilities in the decision-making process, especially regarding the role of the NCU in relation to the regions. A clarification of the roles and responsibilities around current decision-making and accountability processes would help to ensure a common understanding and contribute to enhanced program delivery.
RECOMMENDATION #3: It is recommended that current processes for defining environmental indicators for projects and for measuring, recording and using performance information be assessed to improve the ability of the program to demonstrate its results.
It is difficult to capture the full performance story of the program because the performance measures and the processes for capturing these measures are weak. The EcoAction program currently uses 56 different indicators to measure results for projects, which are too numerous and diverse to add much understanding on the achievement of program outcomes. The difficulty in demonstrating program performance is further impeded by the way the data are gathered and recorded: the MIS, the central housing system for project information, is updated infrequently and data are missing on key project elements that would support the performance story. In addition, relatively little is known about projects that continue after EcoAction funding ends other than information from a few survey questions and some references in final reports submitted at project-end, despite the fact that one of the aims of the program is to encourage organizations to build sustainable projects. Added together, these issues ultimately affect the program’s ability to demonstrate the longer-term impacts and benefits of funding sustainable community projects.
The ES Board agrees with these three recommendations.
The EcoAction Community Funding Program is one of Environment Canada’s twelve Community Action Programs for the Environment (CAPE). In Fall 2007, the CAPEE, which focused on simplifying and streamlining the administration of federal grants and contributions. Various activities that are already under way, or will be initiated as part of our Optimization Initiative and Environment Canada’s new Action Plan for G&C Reform, will support the EcoAction Community Funding Program in responding to the recommendations of this evaluation. Specifically, these initiatives involve:
It is important to note that the scope and timelines of the commitments in relation to the Optimization Initiative and the Departmental Action Plan for G&C Reform are outside of the program’s control.
The program acknowledges that the EcoAction website is difficult to access from the Environment Canada home page. The one-window approach of the CAPE web portal, under the Optimization Initiative, will make it easier to find the EcoAction website and will also provide common tools and resources to assist funding applicants. To address funding applicants’ difficulties in understanding the project eligibility criteria, the program will provide a more complete list of eligible and ineligible project activities that will allow for greater program transparency and clarity. Significant improvements were made to the program forms and tools in 2008, based on clients’ feedback and input. The Evaluation findings, which are based on the 2008 survey of EcoAction funding recipients, may not accurately reflect these changes since they were implemented afterwards. However, there is room for improvement and further steps will be taken to make the EcoAction website and program information easy to use and understand.
EcoAction is a national funding program delivered through Environment Canada’s five regional offices. The national coordination unit is located in the National Capital Region (NCR). We agree that we need more clearly defined roles and responsibilities for the various members of the program team and clear processes to ensure national consistency in the delivery of the program. The program currently holds monthly management team and project officer conference calls, and has ad hoc working groups, to provide the opportunity to discuss and take action on delivery processes and best practices. All program delivery modifications are reviewed and considered by the management team and when appropriate, integrated into the program Operational Guidelines.
The roles and responsibilities around decision making have shifted significantly in the Department over the past few years from the traditional organizational structure to the more recent results-management structure. This caused a great deal of confusion for program managers and staff during the period covered by this evaluation. These shifts in departmental organizational structure have created a sense of uncertainty around the role of the National Coordination Unit (NCU) and about who is responsible for decision making. As the Department is now shifting back to the traditional organizational structure, authority for project funding approvals has gone back to the responsible Regional Directors General (RDG). Program management and design decisions remain a collaborative process involving regional participation and NCU.
EcoAction delivers on a wide range of environmental issues that relate to the four key priorities of the Department: climate change, clean air, clean water and nature. EcoAction projects also result in social, economic and capacity-building benefits to communities. Thus, since 1995, an extensive list of indicators has been developed. The program acknowledges that this list needs to be reduced, focusing on those indicators most relevant to program outcomes and departmental reporting priorities and performance. In addition, EcoAction’s Management Information System (MIS) has been in place since 1998 and has served the program well to capture project information, automate administration processes, and provide detailed reports. However, we recognize that there are inconsistencies in how the data are inputted and uncertainty surrounding possible future expansion of the database to accommodate new needs and pressures. A national MIS working group has been established to assess our current data management process.
The program commits to the following actions in response to the three recommendations:
DATE | ITEM |
---|---|
March 2010 |
|
March 2011 |
|
March 2012 |
|
DATE | ITEM |
---|---|
March 2010 |
|
March 2012 |
|
DATE | ITEM |
---|---|
March 2010 |
|
March 2011 |
|
March 2012 |
|
Contact person: EcoAction National Manager