This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Skip booklet index and go to page content

Evaluation of Freshwater Programs under the Action Plan for Clean Water

Evaluation Design

2.1 Purpose and Scope

The evaluation of the Freshwater Programs addresses Treasury Board requirements for program renewal and Financial Administration Act (amended by the Federal Accountability Act) requirements for evaluation of grants and contributions every five years (as all three initiatives include a contribution component).

The timing of the study is consistent with the original plan for evaluation for the Lake Winnipeg and Lake Simcoe initiatives, but represents an acceleration of the evaluation timing for the Great Lakes initiative. The evaluation covers Freshwater Programs activities from the first year of funding, 2008–2009, through to mid-2010–2011 and examines the following two key areas, as per the 2009 TBS Evaluation Policy.

  • Relevance
    1. Is there a continued need for the Freshwater Programs?
    2. Are the Freshwater Programs aligned to federal government priorities?
    3. Are the Freshwater Programs consistent with federal roles and responsibilities?
  • Performance
    1. To what extent have intended outcomes been achieved as a result of the Freshwater Programs?
    2. Are appropriate performance data being collected, captured, and safeguarded? If so, is this information being used to inform senior management/decision-makers?
    3. Is the Freshwater Programs’ design appropriate for achieving expected results?
    4. Have there been any unintended (positive or negative) outcomes?
    5. Are the Freshwater Programs undertaking activities and delivering products in the most efficient manner?
      • How could the efficiency of the Freshwater Programs’ activities be improved?
      • Are there alternative, more efficient, ways of delivering the Freshwater Programs?
    6. Are the Freshwater Programs achieving their intended outcomes in the most economical manner?
    7. What lessons have been learned at the program and/or initiative levels? Have they been used to improve program performance and/or results-based management? Have they been instructive to the design and/or improvement of other programs?
    8. What are the most important opportunities for improvement going forward with the Freshwater Programs?

The complete matrix of evaluation questions, indicators and data sources is presented in Appendix B (under separate cover).

2.2 Evaluation Approach and Methodology

Multiple lines of evidence were used to increase the reliability and robustness of the evaluation of the Freshwater Programs. The evaluation included a review of documentation and literature, a review of contribution agreement files for the Lake Winnipeg and Lake Simcoe contribution funding programs and project management reports for the GLSRP, key informant interviews, and case studies. Each of the methods is described below.

2.2.1 Document and Literature Review

Secondary sources were reviewed to develop a detailed profile of the Freshwater Programs and to contribute to addressing several of the evaluation questions, including relevance and program performance (achievement of program outcomes and the cost-efficiency analysis, in particular) as well as an analysis of program alternatives. Sources were identified through an Internet-based search, as well as by key informants, evaluation steering committee members and Environment Canada program personnel, and included documents such as background/planning and financial materials; program guides, program management documents, and performance documents (e.g., Evaluation Framework for the Freshwater Programs, Evaluation of the Great Lakes Action Plan IV, Great Lakes International Joint Commission (IJC) reports); corporate/ policy documents; program workplans/progress reports; and other material. A document review template was developed to summarize findings in the documents pertaining to the evaluation questions.

2.2.2 File Review

A sample of 55 contribution agreements funded under the Lake Winnipeg Basin Stewardship Fund and the Lake Simcoe Clean-Up Fund were reviewed to examine outputs and outcomes of the LWBI and LSCUF at the project level. For Lake Winnipeg, all completed projects funded in 2009–2010 and 2010–2011 were reviewed. For Lake Simcoe, a sample of files was reviewed. The review of files was distributed as follows:

  • 13 files for contribution agreements funded under the Lake Winnipeg Basin Stewardship Fund (or about 40% of files); and
  • 42 files for contribution agreements funded under the Lake Simcoe Clean-up Fund (or about 30% of files), stratified by year and agreement size to be representative of the population of contribution agreements.
  • Only closed project files were included in order to review outputs and outcomes as indicated in the project final reports.

A data collection template was used to capture file information in a consistent manner, and to ensure that the content of the files was well documented and linked to the specific evaluation questions and indicators (see Appendix G under separate cover). The file review addressed issues related to the alignment of contributions to departmental objectives, the achievement of outputs and outcomes, unintended impacts, and project costs and leveraging. A profile of the characteristics of the files that were reviewed is included in Appendix H (under separate cover).

The contribution agreement files were generally well documented. Core or mandatory documents such as contribution agreements and proposal assessment documents were available consistently. Other documents that are produced depending on the nature of the project were less consistently available in the files than would be expected. Contents typically included the contribution agreement (98% of files), project proposal (96% of files), proposal review/technical review information (95% of files), and financial information or records (89% of files). Just over 8 in 10 files (82%) included progress or interim reporting and two thirds of the files contained final reports.

Table 2.1: Distribution of Documents in Freshwater Program Contribution Agreement Files Reviewed
 Files (%)
MOU / contribution agreement98
Proposal review / approvals95
Financial files / records89
Progress reporting / activity reporting82
Final report67
Letters of support39
Maps / photos39
Letters of intent (LOIs)38
Permits / plans27
Presentation / reports / summaries22
Site visit reports20
Annual report18
Media / promotional material7
Audit / evaluation reporting6
Meeting / committee agendas / minutes2
Other (e.g., technical assessments)10

The GSLRP funding activity differs from the grants and contributions funding of the LWBSF and LSCUF. Thus the file documentation was not in a format that could be captured by the file review template. As a result, a separate and qualitative review of project management reports for each of the AOCs was conducted to contribute to an understanding of the status of sediment remediation projects.

2.2.3 Key Informant Interviews

In total, 39 key informant interviews were completed with EC personnel, technical and public advisory committee members, other government representatives, project proponents (funded and unfunded) and external experts. Interviews were selected as a line of evidence to obtain the views of those most knowledgeable about the program, or who are beneficiaries of the program. The distribution of interviews was as follows:

  • senior departmental managers, program managers and staff involved in the delivery of the Freshwater Programs (11);
  • federal and provincial representatives for each initiative (7);
  • public advisory committee members (4);
  • contribution funding project proponents (9);
  • unfunded applicants (4); and
  • external professional and academic experts on freshwater/aquatic ecosystem remediation (4).

The breakdown of interviews by program level and component level is as follows:

  • Freshwater Programs (senior managers): 3
  • Lake Winnipeg: 15
  • Great Lakes: 5
  • Lake Simcoe: 12
  • external: 4

The interviews were conducted by phone. A master key informant interview guide is included in Appendix I (under separate cover). This guide was further customized for each respondent group.

The use of percentages to represent the qualitative information obtained from the key informant interviews is not appropriate. However, to ensure a common understanding of the terms used in the analysis and reporting of interview results, the following guidelines have been used:

  • “A few/a small number of interviewees” = less than 25%;
  • “Some/a minority of interviewees” = 25 to 49%;
  • “A majority of interviewees” = 50 to 75%;
  • “Most interviewees” = over 75%; and
  • “Almost all interviewees” = 95%.

2.2.4 Case Studies

In total, nine case studies of Freshwater Programs funded activities were conducted--three for each of the Freshwater component programs--to provide illustrative evidence on the outputs and outcomes of the contribution funding programs at the project level. For the LWBSF and the LSCUF, projects were selected to represent various program activity areas:

  • public education/awareness;
  • science/research and monitoring;
  • point source pollution projects; and
  • non-point source pollution/habitat projects.

The case studies collected qualitative and, where possible, quantitative data to explore project results and their contribution to the Freshwater Program’s immediate and intermediate outcomes. The case studies included the following:

  • a review of project file information, such as the project proposal/funding data, financial and administrative data, performance/ecosystem data and other profile data;
  • a review of other project documentation, such as project / departmental documents, reports and communications, outputs, interim or evaluation reports; and
  • key informant interviews (n=2-3) with project proponents, an EC representative, provincial/federal representative(s) and/or other key stakeholder(s) (The case study interview guide is included in Appendix J, under separate cover).

Case study summaries were prepared based on the documentary sources and interviews, and are included in Appendix K (under separate cover). Key observations and illustrative examples from the case studies have been woven throughout the integrated reporting as appropriate.

2.3 Challenges and Limitations

A number of methodological caveats should be noted for this evaluation.

First, much of the information that was gathered for this review is drawn from internal and often qualitative sources--program documentation, as well as interviews with program managers or federal partners, committee members and project proponents who have a clear stake in the program. This potential bias was mitigated to some extent by including unfunded contribution fund applicants and external experts as interview respondents.

Second, the Freshwater Programs are not a program per se, but rather a collection of three quite unique component programs (3 distinct sub-activities) targeted to distinct watersheds. While there have undoubtedly been efficiencies in the evaluation in grouping the programs, evaluation resources have been thinly spread across these three programs. The evaluation findings (and conclusions and recommendations) are framed at the Freshwater Programs level where appropriate, though many of the findings did not converge and so are presented at the level of the component programs.

Finally, the achievement of program objectives of restoration/remediation in the lakes is difficult to measure and to assess the program contribution. This is due, in part, to the complex ecological issues at play in these water bodies, the long-term nature of their resolution, as well as the influence of external factors (e.g., the availability of funding for infrastructure projects and regulatory policies are critical to the achievement of program outcomes). Thus, evaluation findings related to the ultimate outcomes of Freshwater Programs must be considered in this context.

Date modified: