Evaluation of the EcoAction Community Funding Program

May 2009

| ToC | Next |


EXECUTIVE SUMMARY

Environment Canada’s Evaluation Division, Audit and Evaluation Branch, conducted an evaluation of the EcoAction Community Funding Program. The evaluation was identified as part of the Departmental Audit and Evaluation Plan approved in April 2008 and was conducted as a requirement for program renewal, focusing on the fiscal years from 2004–05 to 2008–09.  The EcoAction program will be seeking renewal for its terms and conditions in the spring of 2010-11.

EcoAction focuses on supporting community-level projects led by non-profit organizations that are expected to have measurable, positive results for the environment.  While national in scope, the program is managed in the regions through a network of Environment Canada offices across Canada.  The objective is to offer project support to EcoAction clients throughout the application process and to monitor funded projects closely.  

Eligible projects may be funded up to a maximum of $100,000 per project which, by design, makes EcoAction a funder of small projects (the average funding is about $25,000 per project).  In order to be funded, applicants must lever at least 50% of the total value of the project from sources other than the federal government.  The maximum duration for a funded project is two years and projects that are designed to become self-sustaining after this period are encouraged.  

The objectives of the EcoAction program are to:

  1. enable community-based groups to achieve environmental results related to departmental priorities and thereby reduce risks to human health and the environment;
  2. lever in-kind and monetary support from non-federal government sources for environmental activities that have measurable environment benefits; and
  3. provide Canadians with the tools they need to act on their knowledge and values as individuals and members of communities in support of sustainable development.

Funded projects must have positive intended results for the environment in one of four priority areas of the EcoAction program: Clean Air, Climate Change, Clean Water, and Nature.

EVALUATION ISSUES

The Evaluation of the EcoAction Community Funding Program assessed the relevance, success and cost effectiveness, and to a lesser extent revisited the design and delivery, of the EcoAction program. The evaluation was designed to determine whether the program:

METHODOLOGY

Data were collected for the evaluation using multiple lines of evidence. These included a document/file review, secondary data analysis of information recorded in the Management Information System (MIS), analysis of the 2008 client survey (n = 126), 18 key informant interviews with staff and stakeholders, and a survey of non-funded applicants (n = 154). Notwithstanding the strengths related to using multiple lines of evidence, some challenges and limitations were encountered, including difficulties in attributing findings to the EcoAction program and inconsistencies in MIS data.

EVALUATION FINDINGS

Overall, evaluation findings indicate that EcoAction provides an important source of support for community-based environmental projects, as described below.  However, the out-of-date program data in the MIS and the limited data available on each of the program’s indicators make it difficult to show program results and determine the achievement of outcomes.

Evaluation findings are summarized in the following sections by evaluation issue.

a) Relevance

EcoAction supports community-level, environmental projects that reflect Environment Canada priorities, such as the reduction of greenhouse gas (GHG) emissions, improvements in air and water quality and protecting the species and their habitat.  Those involved in the program or projects report that without EcoAction, a variety of projects would not exist. Some of the evaluation’s key findings include: 

b) Success

Performance information on projects funded through EcoAction should provide a clear indication of whether or not the intended outcomes of the program have been achieved.  However, analyses for this evaluation were limited by the quality of the data available. These data are largely based on the opinions of program staff and EcoAction clients, often not quantified in terms of environmental outcomes, and not validated. The following are key findings:

c) Cost Effectiveness

The cost effectiveness of the EcoAction program was difficult to assess because accurate program expenditures (both operating and salary) are difficult to determine and because of limitations in measuring achieved outcomes.  As a result, cost effectiveness was examined indirectly by assessing the efficiency of the EcoAction program.  Findings indicate that, while the program’s design necessitates high administrative costs, some improvements may be made to improve the cost efficiency of the EcoAction program.

d) Design and Delivery

It should be noted that this is an impact evaluation, conducted as a requirement for program renewal and was limited in scope to explore design or delivery issues.

RECOMMENDATIONS

Three recommendations were developed for the Ecosystem Sustainability (ES) Board based on the evaluation findings and conclusions.

RECOMMENDATION #1: It is recommended that program forms and tools be improved to make them easier to understand and to use.
Evaluation findings show that, while clients were satisfied with the services received by EcoAction staff, they were less satisfied with the ease of understanding of program forms and tools.  In particular, funding applicants noted difficulties navigating the EcoAction website, understanding the application guide and eligibility criteria as well as completing the application form.  Funded applicants also identified difficulties completing reporting forms used to monitor their projects.  

RECOMMENDATION #2: It is recommended that roles, responsibilities and processes be examined to identify opportunities for clarification and increased efficiencies.  
The EcoAction program was designed to fund projects at the community level with an emphasis on client service and this contributes to its high administrative costs to operate the program.  The identification of best practices, particularly in the regions, and areas where streamlining the delivery process might be possible could help to improve the overall efficiency of the program.  As well, the evaluation identified some uncertainty among program staff around roles and responsibilities in the decision-making process, especially regarding the role of the NCU in relation to the regions.  A clarification of the roles and responsibilities around current decision-making and accountability processes would help to ensure a common understanding and contribute to enhanced program delivery.      

RECOMMENDATION #3: It is recommended that current processes for defining environmental indicators for projects and for measuring, recording and using performance information be assessed to improve the ability of the program to demonstrate its results.
It is difficult to capture the full performance story of the program because the performance measures and the processes for capturing these measures are weak.  The EcoAction program currently uses 56 different indicators to measure results for projects, which are too numerous and diverse to add much understanding on the achievement of program outcomes.  The difficulty in demonstrating program performance is further impeded by the way the data are gathered and recorded: the MIS, the central housing system for project information, is updated infrequently and data are missing on key project elements that would support the performance story.  In addition, relatively little is known about projects that continue after EcoAction funding ends other than information from a few survey questions and some references in final reports submitted at project-end, despite the fact that one of the aims of the program is to encourage organizations to build sustainable projects.  Added together, these issues ultimately affect the program’s ability to demonstrate the longer-term impacts and benefits of funding sustainable community projects.  

MANAGEMENT RESPONSE

The ES Board agrees with these three recommendations.

The EcoAction Community Funding Program is one of Environment Canada’s twelve Community Action Programs for the Environment (CAPE).  In Fall 2007, the CAPEE, which focused on simplifying and streamlining the administration of federal grants and contributions.  Various activities that are already under way, or will be initiated as part of our Optimization Initiative and Environment Canada’s new Action Plan for G&C Reform, will support the EcoAction Community Funding Program in responding to the recommendations of this evaluation.  Specifically, these initiatives involve:

It is important to note that the scope and timelines of the commitments in relation to the Optimization Initiative and the Departmental Action Plan for G&C Reform are outside of the program’s control. 

The program acknowledges that the EcoAction website is difficult to access from the Environment Canada home page.  The one-window approach of the CAPE web portal, under the Optimization Initiative, will make it easier to find the EcoAction website and will also provide common tools and resources to assist funding applicants.  To address funding applicants’ difficulties in understanding the project eligibility criteria, the program will provide a more complete list of eligible and ineligible project activities that will allow for greater program transparency and clarity.  Significant improvements were made to the program forms and tools in 2008, based on clients’ feedback and input.  The Evaluation findings, which are based on the 2008 survey of EcoAction funding recipients, may not accurately reflect these changes since they were implemented afterwards.  However, there is room for improvement and further steps will be taken to make the EcoAction website and program information easy to use and understand.

EcoAction is a national funding program delivered through Environment Canada’s five regional offices.  The national coordination unit is located in the National Capital Region (NCR).  We agree that we need more clearly defined roles and responsibilities for the various members of the program team and clear processes to ensure national consistency in the delivery of the program.  The program currently holds monthly management team and project officer conference calls, and has ad hoc working groups, to provide the opportunity to discuss and take action on delivery processes and best practices.  All program delivery modifications are reviewed and considered by the management team and when appropriate, integrated into the program Operational Guidelines.  

The roles and responsibilities around decision making have shifted significantly in the Department over the past few years from the traditional organizational structure to the more recent results-management structure.  This caused a great deal of confusion for program managers and staff during the period covered by this evaluation.  These shifts in departmental organizational structure have created a sense of uncertainty around the role of the National Coordination Unit (NCU) and about who is responsible for decision making.  As the Department is now shifting back to the traditional organizational structure, authority for project funding approvals has gone back to the responsible Regional Directors General (RDG).  Program management and design decisions remain a collaborative process involving regional participation and NCU.  

EcoAction delivers on a wide range of environmental issues that relate to the four key priorities of the Department: climate change, clean air, clean water and nature.  EcoAction projects also result in social, economic and capacity-building benefits to communities.  Thus, since 1995, an extensive list of indicators has been developed.  The program acknowledges that this list needs to be reduced, focusing on those indicators most relevant to program outcomes and departmental reporting priorities and performance.  In addition, EcoAction’s Management Information System (MIS) has been in place since 1998 and has served the program well to capture project information, automate administration processes, and provide detailed reports.  However, we recognize that there are inconsistencies in how the data are inputted and uncertainty surrounding possible future expansion of the database to accommodate new needs and pressures.  A national MIS working group has been established to assess our current data management process.   

The program commits to the following actions in response to the three recommendations:

RECOMMENDATION #1: It is recommended that program forms and tools be improved to make them easier to understand and to use.
DATE  ITEM
March 2010
  • Review and update the Web content and architecture of the EcoAction website to improve navigation and incorporate Government of Canada common look and feel requirements;
  • Publish a detailed list of eligible and ineligible projects for potential funding recipients;
  • Participate in the business process analysis that is part of the Optimization Initiative to determine best practices and streamline and standardize forms and tools used by the program.
March 2011
  • Implement appropriate reporting tool and requirements for funded projects as determined by the departmental G&C risk assessment strategy developed as part of the Departmental Action Plan for G&C Reform. 
March 2012
  • Adopt the departmental online application and information management system, which will enhance client service and facilitate application, monitoring and reporting processes for both clients and program staff.

RECOMMENDATION #2: It is recommended that roles, responsibilities and processes be examined to identify opportunities for clarification and increased efficiencies.
DATE  ITEM
March 2010
  • Based on a review of how the information is shared, how decisions are made and required approvals, define roles and responsibilities at all levels of the program around decision making and accountability;
  • Develop service standards and update Operational Guidelines to better streamline the delivery process overall.
March 2012
  • Adopt the online application and information management system as stated above, which will also contribute to the reduction of administrative burden on clients and staff, and increase efficiencies in program delivery.

RECOMMENDATION #3: It is recommended that current processes for defining environmental indicators for projects and for measuring, recording and using performance information be assessed to improve the ability of the program to demonstrate its results. 
DATE  ITEM
March 2010
  • Reduce the number of project indicators by selecting those that are most relevant to program objectives and departmental outcomes, and align them with indicators included in the Performance Measurement Framework (PMF) that accompanies the Program Activity Architecture (PAA) 2010-11;
  • Develop a user guide to improve consistency of data entry in the program's MIS;
  • Provide training to all program staff on the use of the MIS.
March 2011
  • Establish a verification process to follow up after project completion so as to provide feedback to the program on the longer-term impacts, benefits, and overall sustainability of community projects;
  • Integrate lessons learned and best practices from the review of completed projects into EcoAction's application review and decision-making processes.
March 2012
  • Provide measurement tools to clients to facilitate reporting on results. The program will conduct a review of the existing departmental measurement tools. If required, EcoAction will seek departmental expertise to develop appropriate tools;
  • Continue to conduct a client survey every four years (offset between program evaluations).

Contact person: EcoAction National Manager


| ToC | Next |