This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Evaluation of Environment Canada's Class Grants and Contributions

July 2009

Report Clearance Steps

Planning phase completed - March 2008
Report sent for management response - May 2009
Management response received - May 2009
Report completed - May 2009
Report approved by Departmental Audit and Evaluation Committee (DAEC) - July 2009

abbrs used in the report

ACAP
Atlantic Coastal Action Program
ARAF
Accountability, Risk and Audit Framework
CA
Contribution Agreement
DPR
Departmental Performance Report
EC
Environment Canada
EMC
Executive Management Committee
G&Cs
Grants and Contributions
IS
Internal Services
NGO
Non-governmental Organization
NRCan
Natural Resources Canada
O&M
Operations and Maintenance
OPG
Outcome Project Group
OPP
Outcome Project Plan
OPSC
Outcome Project Sub Component
RMS
Results Management Structure
TBS
Treasury Board of Canada Secretariat

Acknowledgments

The Evaluation Project Team, including Robert Tkaczyk and Karine Kisilenko, led by William Blois under the direction of the Director, Shelley Borys, would like to thank all those individuals who contributed to this project and particularly all interviewees who provided insights and comments crucial to this evaluation. The evaluation team would also like to thank Janice Remai from Ekos Research Associates, whose team was contracted to conduct the administrative file review.

Prepared by the Evaluation Division, Audit and Evaluation Branch

Table of Contents

EXECUTIVE SUMMARY

In 2008-09, Environment Canada's Evaluation Division, Audit and Evaluation Branch, conducted an evaluation of Environment Canada's Class Grants and Contributions for the 2006-07 and 2007-08 fiscal years. This project was selected for inclusion in Environment Canada's Risk-Based Audit and Evaluation Plan 2008-2011, approved by the Departmental Evaluation Committee in May 2008, as a precondition for the renewal of the terms and conditions of the Class Grants and Contributions.

Environment Canada (EC) Grants and Contributions (G&Cs) are used to transfer monies from the federal government to individuals, organizations (e.g., academic institutions, community organizations, non-governmental organizations) or other levels of government (e.g., provincial governments), with the purpose of advancing government policy and departmental objectives. While traditional G&C programs are developed to provide support in very specific program areas or initiatives, this "class management" approach supports grants and contributions that are designated to a broad theme rather than to a specific program or initiative, thereby simplifying the management of G&Cs and providing the Department ongoing flexibility to respond in a timely manner to new opportunities.

The class G&Cs consist of overarching classes which are divided into one grant and three contribution authorities and are also used to support the G&C components of larger programs, known as "sub-class" programs. These sub-class programs are stand-alone programs which have secured individual Treasury Board authority, but do not have individual G&C terms and conditions since the Class terms and conditions provide the needed authority.

Collectively, the broad objectives of the class G&Cs are to:

  • ensure a broad knowledge base exists to support EC science and technology needs;
  • stimulate research and the development of mechanisms for co-ordinating and disseminating research-related information;
  • encourage environmental and sustainable development initiatives at the regional or ecosystem level; and
  • ensure that Canada's interests are represented at international fora.

EVALUATION ISSUES

The Evaluation of Environment Canada's Class Grants and Contributions assessed the relevance, success, cost-effectiveness, and design and delivery of the classes. The evaluation was designed to determine whether the classes are:

  • aligned with and contributing to federal government priorities, and whether they address actual needs (relevance);
  • achieving or on track to achieving their intended outcomes (success);
  • using the most appropriate and efficient means to achieve their outcomes (cost-effectiveness/alternatives); and
  • designed and delivered in the best possible way (design and delivery).

METHODOLOGY

Data were collected for the evaluation using multiple lines of evidence. These included a document/file review, a review of grant and contribution agreement files (n=104), and a total of 41 key informant interviews, conducted with departmental program managers (n=12), representatives of the Finance and Corporate Branch (n=4), and recipients of class grants or contributions funding (n=25). Key challenges which had to be overcome for the evaluation included an inability to link all class agreements to a small set of shared outcomes and the absence of a management information system to provide centralized tracking of both financial and other documentation in support of G&C agreements.

EVALUATION FINDINGS

Overall, evaluation findings indicate that the classes are an important source of support to departmental stakeholders and partners in pursuit of activities aligned with departmental priorities. The absence of a centralized management information system and unclear program-level accountabilities, however, have impeded performance measurement, accountability and reporting. Complex approval and financial review processes may also have diminished the efficient delivery of the classes.

Evaluation findings are summarized in the following sections by evaluation issue.

a) Relevance

Environment Canada's class G&Cs continue to be an important means of supporting third parties in the pursuit of departmental priorities. Unlike conventional G&C programming, the broad scope of classes provide managers across the Department access to important tools (grants and contributions) to further departmental objectives and to respond quickly to opportunities and changing priorities.

Class agreements tend to be well aligned with departmental priorities and are appropriately targeted to a variety of audiences reflecting the broad mandate of the Department. Numerous tools and processes have been established to ensure that agreements are well aligned with priorities, although there is a need to update some tools for managers to ensure they remain consistent with the Department's changing structure. There is no evidence to suggest that the classes are used to undertake activities for which programs are directly responsible or to avoid the development of unique G&C authorities.

b) Success

The intended results of agreements funded through the classes have for the most part been achieved, although this was not always evident for ongoing projects, in cases where funding delays negatively affected the achievement of project results or when other barriers (e.g., technological issues, environmental circumstances) were present. Performance-measurement information was most likely to be output-based, although for many agreements (e.g., those with longer-term outcomes, outcome information not stipulated/appropriate, or that present measurement challenges) information related to outputs is all that could reasonably be expected.

The classes were instrumental in the achievement of project results in a large majority of cases. These class authorities were often either the only funding source, a catalyst to leverage resources, or sufficiently flexible to allow the project to proceed as designed and/or be adjusted as circumstances warranted.

The classes are widely regarded as contributing to the achievement of departmental objectives, most notably because the classes leverage resources (e.g., partners, expertise) and provide the flexibility to support new priorities, a range of activities and be strategic in directing funding. The impact of the classes on the achievement of departmental priorities has been dampened somewhat by a long and complex agreement-approvals process.

c) Cost-Effectiveness

Administrative costs and overall projects costs were not available to assess directly the costs of delivering the classes. Nonetheless, the class G&Cs are generally felt to be the most appropriate and/or cost-efficient means to support stakeholders in the pursuit of departmental priorities when compared to any of the other tools that exist to support third parties (e.g., contracts, unique G&C authorities, partnership agreements). The flexibility of the class G&Cs to respond to opportunities, adjust to changing priorities, and re-allocate funding was the primary reason most respondents felt this way, as these characteristics of the classes allow the Department to reduce administrative burden, leverage more funding and avoid lapsing funds.

Suggestions to improve the efficiency of the classes related primarily to the implementation of a more balanced risk-mitigation approach to their delivery, including more timely approvals and funding, delegating some authorities for approvals, clarifying roles and responsibilities for their delivery (and so avoiding duplication), balancing flexibility and accountability, and engaging in multi-year agreements.

A minority of respondents were of the view that relative to unique G&C programs, evaluation, accountability and reporting are easier under the classes as a result of the nationally consistent approach and expertise that has evolved for their management. More frequently, however, the broad nature of the classes was felt to diminish the ability to evaluate and report on them (i.e., outcomes are too disparate to be sensibly aggregated) or to assign clear accountability for the classes. The absence of a centralized information-management system for the classes further exacerbates the difficulties of evaluating these class authorities and measuring their performance.

d) Design and Delivery

There is general agreement that the classes are being delivered as designed, as they provide flexibility to respond to priorities (felt to be a key aspect of the classes' design) and are well delivered to recipients. Some concern was voiced about the lack of consistency with which different processes (e.g., governance, risk management, performance measurement, reporting) are implemented across departmental regions, and there may be room to improve oversight of funded projects in terms of proposal reviews and output/outcome reporting.

Although managers and finance personnel generally expressed a strong familiarity with most existing processes for the management and monitoring of the classes, the project approvals process was perceived to be burdensome and inefficient, and to be causing delays that may impede project success, hurt stakeholder relations and reduce the flexibility of the classes to respond to priorities.

Environment Canada has developed tools to ensure that class contribution agreements identify indicators, deliverables and expected results. Evidence from key informants and the file review suggest that they are nearly always identified, although in some cases they may not have been required or appropriate. The nature of the indicators identified varied and was more often linked to output-based information rather than on outcomes, although both types of indicator were frequently identified.

Tools and guidelines have also been established within the Department to ensure that performance information is collected and reported for class agreements. Evidence from key informants and the file review suggest that this information is being collected for most agreements, although the nature of this information varies between output- and outcome-based information, depending on the nature of the agreement. While performance reporting was not a requirement of the grant or contribution agreement in some cases, such information was not available for one in five agreements reviewed and for which it was reasonable to expect performance reporting.

Although performance information is consistently reported at the agreement level, the broad and heterogeneous nature of the class agreements meant that no systematic process for aggregating or reporting this information to senior decision-makers was in evidence. Given the original intent of the class management approach (i.e., to simplify management of G&Cs and provide flexibility to respond to opportunities), the absence of a systematic performance reporting process at the class level is not surprising.

Accountabilities related to the approvals, financial oversight and management of class agreements are clearly articulated and documented, and are generally understood by most internal key informants. The broad nature of the classes meant that accountabilities for overall class results are much more difficult to identify.

RECOMMENDATIONS

Six recommendations were developed for the IS Board based on the evaluation findings and conclusions.

  1. The IS Board should clarify overall accountabilities for the class G&Cs. One option to achieve more clarity in this regard would be to re-align the classes according to the Department's new program activity architecture (PAA) and strategic outcomes. In this way, overall accountability for each of the classes would rest with the senior manager who wields primary responsibility for the departmental activities that support each of the Department's strategic outcomes.
  2. The IS Board should clarify the respective roles, responsibilities and accountabilities of managers and finance personnel in the financial review, management and oversight of funded agreements. Greater clarity may in turn lead to process efficiencies by avoiding duplication in initial financial approvals and the management of project deliverables and payments.
  3. The IS Board should explore means of streamlining the G&C approvals process. To this end, consideration should be given to making the process more risk-based, providing early approvals for less discretionary G&C spending, implementing delegated authorities and incorporating the use of multi-year agreements where appropriate.
  4. The IS Board should develop a more strategic approach for ongoing performance measurement of class G&Cs.This approach should consider linking unique project outcomes, activities and/or outputs with shorter-term departmental outcomes to which an attributional link is more easily made and to more clearly outline each project's logical association with the Department's strategic outcomes.
  5. The IS Board should update standardized tools for managers to ensure they are aligned with the current departmental structure, reflect the needs of the new Policy on Transfer Payments, and are employed in a consistent manner across programs and regions. In updating these tools, the IS Board should consider:
    • the development of a standard training module for managers wishing to make use of classes in order to improve the quality of agreements and so reduce the burden on finance personnel and improve the efficiency of the financial review process;
    • language which encourages the identification of clear and measurable outcomes for performance measurement purposes in instances where the collection of outcome-related information is feasible within the timeframe and financial scope of funded projects; and
    • the development of standard indicators to guide managers and project proponents in the measurement of project success.
  6. The IS Board should develop a centralized information-management system to track financial and performance information for class G&C agreements, to facilitate the aggregation and reporting of this information to senior departmental decision-makers, and to improve the accessibility and comprehensiveness of this information for evaluation.

MANAGEMENT RESPONSE

The ADM, Finance and Corporate Branch, and the ADM, Environmental Stewardship Branch, (the ADMs) take responsibility on behalf of all the involved branches for implementing the management response.

The proposed responses to the recommendations draw from the draft EC Action Plan for G & C Reform, which will be presented for approval to EMC on May 27th. It is proposed that the responses outlined below be implemented department-wide (not just by IS Board) as all four boards use classes to fund projects.

Six recommendations were developed for the IS Board based on the evaluation findings and conclusions.

1. The IS Board should clarify overall accountabilities for the class G&Cs. One option to achieve more clarity in this regard would be to re-align the classes according to the Department's new program activity architecture (PAA) and strategic outcomes. In this way, overall accountability for each of the classes would rest with the senior manager who wields primary responsibility for the departmental activities that support each of the Department's strategic outcomes.

The ADMs agree with this recommendation.

Accountabilities for Class G&C programs will be clarified through the development of new Class G&C authorities. It is proposed that these new Classes be aligned to the department's new program activity architecture (PAA) and strategic outcomes. Through this alignment, the operational Boards are responsible for recommending to the Deputy Minister program priorities and the annual budget allocation under the new classes and for monitoring the overall achievement of results. Branch heads and their delegates, supported by Corporate Finance, are responsible for the negotiation, implementation and achievement of results of individual agreements under the new classes.

Responsible Party: Co-lead, ADMs of ESB and Finance; Coordination of the Renewal process will be the responsibility of Board Secretaries (ES/EP/WES)

2. The IS Board should clarify the respective roles, responsibilities and accountabilities of managers and finance personnel in the financial review, management and oversight of funded agreements. Greater clarity may in turn lead to process efficiencies by avoiding duplication in initial financial approvals and the management of project deliverables and payments.

The ADMs agree with this recommendation.

This was also a recommendation of the "Angiogram" work undertaken in fiscal year 2008-09, and will be a key deliverable under Environment Canada's Action Plan for G&C Reform. An important step has already been taken in support of this recommendation with the approval of delegation of authorities by the Minister in March, 2009. Led by Corporate Finance, the ADMs will clarify roles and responsibilities of managers and finance personnel in the development of contribution agreements and the subsequent financial review, sign-off and oversight. A clear, documented and streamlined process will be the result. Training and information for managers and finance officers will ensure the process is widely understood and consistently applied across the department.

In particular, Corporate Finance will:

  • Develop information sessions for G&C program managers and finance officers on: Treasury Board's new Policy on Transfer Payments; EC's G&C Approval Process and EC's Action Plan on G&C Reform
  • Prepare technical training/coaching for MFS's and program managers on new approval processes and updated finance procedures (including checklists and a "G&C 101" internal website)

Expected Completion date: August, 2009 (for clarification of roles & process)
Responsible Party: Lead - Corporate Finance, informed by a departmentally-representative team

3. The IS Board should explore means of streamlining the G&C approvals process. To this end, consideration should be given to making the process more risk-based, providing early approvals for less discretionary G&C spending, implementing delegated authorities and incorporating the use of multi-year agreements where appropriate.

The ADMs agree with this recommendation.

The work referred to in the Management Response to evaluation Recommendation 2 to clarify and streamline the roles and responsibilities of managers and finance personnel in the development, financial review, sign-off and oversight of contribution agreements, will also deal with the initial allocation process and timing of funding decisions and is a key deliverable under the fifth theme of Environment Canada Action Plan for G&C Reform (Achieving Clarity and Consistency in Practices).

Specific deliverables that will respond to this recommendation include the creation and dissemination of step-by-step written guidelines for approval of agreements (which should reduce the number of signatures required), the creation of a standard contribution agreement made available on the internal G&C website, and an update EC Managers' Guide to Gs&Cs. This work to streamline approvals may also be informed by EC's move to a more risk-based approach.

This should result in shorter wait times from application through to approved contribution agreement and reduce frustration by employees and funding recipients due to delays and inefficient management practices

An important step forward was taken for fiscal year 2009-10, when notional G&C budgets for "fenced funds" were allocated to Boards in January, 2009. This has resulted in much earlier allocations of G&C budgets down to the program level. As mentioned above, G&C approvals have also been streamlined with the approval of the delegation of authorities in March, 2009.

The recommendation also refers to the need to build a risk management approach to the approval of Gs&Cs. While a Risk Management framework, along with a suite of risk tools, will be an important theme under EC's Action Plan, the goal of the work is to reduce administrative burden (where appropriate) and not to streamline approval processes.

Expected Completion date: Fall, 2009
Responsible Party: Lead - Corporate Finance, supported by Outreach Division, Environmental Stewardship Branch, and informed by a departmentally-representative team

4. The IS Board should develop a more strategic approach for ongoing performance measurement of class G&Cs. This approach should consider linking unique project outcomes, activities and/or outputs with shorter-term departmental outcomes to which an attributional link is more easily made and to more clearly outline each project's logical association with the Department's strategic outcomes.

The ADMs agree with this recommendation.

The new Class Gs&Cs will be aligned to the department's Program Activity Architecture and Strategic Outcomes, thereby facilitating better performance measurement. Under the new Transfer Payments Policy, a Performance Measurement Strategy (which replaces the RMAF/RBAF requirement of the past) will be developed for each Class. These Strategies will be aligned with the Performance Measurement Framework (PMF) for the PAA Program Activities that are supported by these new class authorities. Each class PMS will outline the logical association between the range of activities potentially supported through the classes and lower-level results and indicators aligned with Program Activity results and to which class agreements can more easily be linked. These common lower-level performance indicators which may be used to measure achievement of results will be used wherever feasible.

In addition to performance measurement at the Class level, Contribution Agreements will specify expected deliverables for each individual project and wherever possible will also identify any measurable project results (i.e., outcomes) that are linked to PAA results (i.e., through common lower-level results identified through the PMS). Project deliverables will provide the basis for payment to the recipient.

As part of the criteria for project selection, the program will ensure that projects are chosen for which there is a strong linkage between the expected project deliverables and/or results and the expected results of the overall PAA program activity. Applicants will be asked to explain in their funding applications what these linkages are and program managers will assess the strength and validity of the proposed linkages based on their knowledge of the subject matter.

Expected completion date: March, 2010
Responsible Party: Co-leads - ADMs Finance and ESB for Renewal of the Classes; Coordination of the Renewal process will be the responsibility of Board Secretaries (ES/EP/WES)

5. The IS Board should update standardized tools for managers to ensure they are aligned with the current departmental structure, reflect the needs of the new Policy on Transfer Payments, and are employed in a consistent manner across programs and regions. In updating these tools, the IS Board should consider:

  • the development of a standard training module for managers wishing to make use of classes in order to improve the quality of agreements and so reduce the burden on finance personnel and improve the efficiency of the financial review process;
  • language which encourages the identification of clear and measurable outcomes for performance measurement purposes in instances where the collection of outcome-related information is feasible within the timeframe and financial scope of funded projects; and
  • the development of standard indicators to guide managers and project proponents in the measurement of project success.

The ADMs agree with this recommendation.

As part of EC's Action Plan for G&C Reform a number of tools are being either updated or created to reflect the requirement of the Transfer Payments Policy (TPP)

Under the Action Plan's Theme 2 (Culture Change) this will include:

  • Information sessions for G&C project officers & administrative support staff on G&C Management. This will reflect, among other updates, EC's Action Plan for G&C Reform, the TPP and the new, streamlined G&C Approval Process for managers.
  • Technical training has aleady been provided to Finance G&C Experts and is being followed up with bi-weekly interactive updates.
  • Training sessions on Risk Management approach (see below)
  • Regular messaging on goals and progress of G&C Reform for Management Council

Under the Action Plan's Theme 4, this will include:

  • A standardized risk management approach and suite of risk assessment tools developed in collaboration with front-line managers

Under the Action Plan's Theme 5, this will include:

  • Step-by-step written guidelines for approval of agreements
  • Standard contribution agreement (template)
  • Updated "EC Managers' Guide to Gs&Cs"
  • Written procedures for direct deposit, cash advances
  • Standard application forms, reports and audit requirements

The ADMs will seek the advice and input of Audit and Evaluation and Corporate Management Directorate to include in these tools, language which encourages managers to include clear and measureable indicators and outcomes in the development of agreements.

Expected completion date: Fall, 2009 for initial round of info sessions & tools; March, 2010 for training; Training of Risk Management tools will begin in 2010-11, following a pilot of the tools in 2009-10
Responsible party: For themes 2 & 5 - Corporate Finance, supported by Outreach Division, Environmental Stewardship Branch; for theme 4 - Co-lead of Outreach Division, Environmental Stewardship Branch and Corporate Finance

6. The IS Board should develop a centralized information-management system to track financial and performance information for class G&C agreements, to facilitate the aggregation and reporting of this information to senior departmental decision-makers, and to improve the accessibility and comprehensiveness of this information for evaluation.

The ADMs agree with this recommendation.

A key deliverable under theme 7 of EC's Action Plan for G&C Reform (Improving Recipient Access and Efficiency Using Technology), is the development of a centralised G&C information management system to track financial, project and performance information. This new system will serve funding applicants and recipients, program managers and finance officers. It will be designed to hold information related to all of Environment Canada's grants and contributions projects and expenditures, including those under the new classes and funding programs.

The first step in developing this system is a business process and needs analysis that will be launched in 2009-10 fiscal year, with an initial focus on the established community funding programs. Funding for this first step has been identified as a priority by the ES Board.

As new Classes are developed over this fiscal year, the information management needs of this type of G&C will be incorporated into the business process analysis.

The result of the work this fiscal year will be a business case for the development of a department-wide system. The investment in such a system will be significant and will be presented for consideration to senior management. In assessing options for the new management information system, we will consider the appropriateness of adopting models used by other government departments.

Expected completion date: March, 2010 (Business case); Target for launch of new system would be 2011-12. If the business case is not accepted, the department would continue to operate existing systems, integrating the new Classes and the new risk tools into these systems.
Responsible party: Outreach Division, Environmental Stewardship Branch lead with CIOB and Corporate Finance

1.0 INTRODUCTION

In 2008-09, Environment Canada's Evaluation Division, Audit and Evaluation Branch conducted an evaluation of Environment Canada's Class Grants and Contributions for the 2006-07 and 2007-08 fiscal years. This project was selected for inclusion in Environment Canada's Risk-Based Audit and Evaluation Plan 2008-2011, approved by the Departmental Evaluation Committee in May 2008, given that an evaluation is a requirement for renewal of the Terms and Conditions of the Department's Class Grants and Contributions.

This document presents the findings and recommendations of the evaluation of Environment Canada's Class Grants and Contributions and is organized in the following way. Section 2 provides background information on the Class Grants and Contributions. Section 3 gives an overview of the evaluation design, including the purpose and scope of the evaluation (Section 3.1) and a discussion of the methods used to conduct the evaluation (Section 3.2). Section 4presents the evaluation's findings. Sections 5 and 6 lay out, respectively, the conclusions and recommendations.

2.0 BACKGROUND

2.1 Profile

Environment Canada Grants and Contributions (G&Cs) are used to transfer monies from the federal government to individuals, organizations (e.g., academic institutions, community organizations, non-governmental organizations) or other levels of government (e.g., provincial governments), with the purpose of advancing government policy and the Department's objectives. Transfer payments are used to support specific activities that extend departmental resources by leveraging funds and/or supporting projects where the Department may lack the expertise or share jurisdiction.[1]

While traditional G&C programs are developed to provide support in a very specific program area, in 1999 Treasury Board approved a "class management" approach to G&Cs. Class grants and contributions are transfer payments that are designated to a broad theme rather than to a specific program or initiative.[2] This class management approach, which consolidated many of EC's numerous transfer payments, was initially suggested by the Treasury Board of Canada Secretariat (TBS) in order to simplify and facilitate the management of G&Cs and give the Department ongoing flexibility to respond in a timely manner to new opportunities. The class G&Cs authorities are also used to support the G&C component of larger programs (e.g., Chemicals Management Plan; Great Lakes Sustainability Fund) which are, in essence, "sub-class" programs. These are stand-alone programs that have secured individual Treasury Board authority, but do not have individual G&C terms and conditions, since the class terms and conditions provide the needed authority. They have a more narrowly defined scope and are more or less managed as unique transfer payment programs.

The class G&Cs consist of overarching classes which are divided into one grant and three contribution authorities:

  • Grants to Support Environmental Research and Development (G01) is intended to support research and the development of highly qualified people in scientific areas related to EC's mandate, such as atmospheric study, wildlife, toxicology, and climate change. The main objectives of the class grant are to ensure: a broad knowledge base exists to support EC science and technology needs; an adequate supply of university graduates trained in areas of strategic importance; and academic centres of expertise exist in priority areas.
  • Contributions to Support Environmental Research and Development (G02) emphasizes increasing the effectiveness of federally supported research and capturing the benefits of partnering. Financial support is provided for stimulating research and the development of mechanisms for co-ordinating and disseminating research-related information.
  • Contributions to Support Environmental and Sustainable Development Projects (G03) enables Canadian groups, associations and organizations to become actively involved in environmental and sustainable development initiatives while accommodating regional ecosystem and socio-economic considerations. Contributions enable recipients to plan, manage and complete environmental and sustainable development initiatives at the regional or ecosystem level. This funding also serves to increase awareness and understanding of environmental and sustainable development issues and to encourage environmentally responsible action.
  • Contributions to Support Canada's International Commitments (G05) provides funding to ensure that Canada's interests are represented at international fora; to sustain and enhance Canada's participation in international organizations and multilateral and bilateral discussions; and to strengthen Canada's linkages to the international community regarding global environmental issues.[3]

Class grants and contributions accounted for approximately $19 million or 40% of the roughly $48 million overall departmental expenditure for grants and contributions in 2006-2007.[4] In 2007-08, class program funding accounted for roughly $26 million or 24% of the roughly $107 of grant and contribution spending Department-wide.[5]As shown in Table 2.1, the roughly $45 million in Class G&Cs funding expenditures from April 2006 to March 2008 supported a total of 758 individual grant and contribution agreements and approximately $17 million of this total was disbursed through sub-classes.

Table 2.1 Agreements by Class G&C: 2006-07 to 2007-08*
ClassTotal value of all agreementsTotal value of sub-class agreementsNumber of agreementsAverage value of agreements
* Values presented may underestimate departmental expenditures on class G&Cs funding during this time period as a result of difficulties in reconciling information between departmental financial systems. See Section 2.1.1 below for a more detailed discussion of these issues.
2006-07
G01$91,000N/A18$5,056
G02$2,965,635$633,00039$76,042
G03$14,474,786$6,076,633297$48,737
G05$1,282,762N/A14$91,626
Total for 2006-07$18,814,184$6,709,633368$51,126
2007-08
G01$29,000N/A9$3,222
G02$3,323,909$1,166,99241$81,071
G03$18,909,238$8,750,151311$60,801
G05$4,104,423N/A29$141,532
Total for 2007-08$26,366,571$9,917,143390$67,607
Total 2006 to 2008$45,180,755$16,626,776758$59,605

Under the Financial Administration Act, all grant and contribution programs must be evaluated in terms of their relevance and effectiveness on a five-year cycle. This evaluation requirement includes class grants and contributions. Although originally scheduled to be renewed for 2009-10, the terms and conditions for all four class G&Cs were extended in the fall of 2008 for a period of one year (to March 31, 2010). They are expected to be renewed for 2010-11.

2.1.1 Challenges of Tracking Class G&Cs Spending

In attempting to profile activity and spending through the class G&Cs during the time period examined in the current evaluation, several issues were identified which affect the ability to develop an accurate picture of the Department's use of the class authorities.

Finance personnel rely on the Discoverer reporting system to track all financial transactions related to a given G&C agreement (i.e., quantity committed; quantity encumbered; quantity billed; quantity cancelled). Within the timeframe for the current evaluation, there was a need to update the organizational codes for some class G&Cs agreements to reflect changes that had been made to the departmental results-management structure. In order to do so, however, it was necessary to cancel the agreement amount in the original agreement entry within Merlin and to re-enter the agreement information with the updated codes on a separate line in Merlin. This re-entry of agreement information may have occurred several times for a single agreement. Unfortunately, the Discoverer reporting system is not able to aggregate information from multiple entries for a single agreement to provide the net value of the agreement. Instead, all entries, whether cancelled, billed or encumbered, are simply summed across agreement entries, thus grossly inflating the true amount of G&C spending through the classes, were these figures to be accepted at face value.

A similar issue affecting the accuracy of information generated using the Discoverer system occurs when invoices for class agreements need to be processed after fiscal year-end using the Payment After Year End (PAYE) system. This system is used to process invoices and payments for a given fiscal year that are not received within the fiscal year to which the invoice amount is to be billed. When a PAYE is set up for an invoice, the outstanding amount listed within the original entry for the class agreement is cancelled. This results in a de-commitment of the outstanding funds for the class agreement in the Procurement (PO) module of Merlin, whereby the total amount encumbered for the agreement is reduced by the PAYE amount. This amount is released back to free balance in the General Ledger (G/L) module of Merlin, from where the PAYE is processed. One portion of the total agreement value is thereby captured in the PO module (amounts billed) and the remainder of the agreement value is captured in the G/L module (PAYE). As such, the only means of reconciling and combining the information from both modules to arrive at the total value of the agreement is to do so manually, by reconciling between two different reports from Discoverer.

In addition, there are a few minor issues that occur in rare cases and which may also affect the accuracy of class information derived from existing databases. For example:

  • Where payments or partial payments have already been made against an initial agreement entry within the Merlin database, they may not be reflected in subsequent entries for the agreement. In some cases, these updates may have been corrected solely using journal vouchers and not necessarily through the PO module within Merlin. This means the amounts showing in the "Quantity Billed" column in the Discoverer report may or may not be accurate.
  • The "PO module" upon which the Encumbrance report from Discoverer is based does not capture instances where G&C money was paid out to a recipient, the project was subsequently cancelled and the money returned to EC by cheque.
  • Financial information for agreements is maintained separately for each fiscal year and multi-year agreements are not identified. To identify multi-year agreements, it would be necessary to access the financial system one year at a time to determine whether unique commitment numbers continue across different fiscal years.

2.2 Roles and Responsibilities

Management oversight of the class grants and contributions is provided through the same governance framework used for all G&C programs in the Department and includes Ministerial, Executive Management Committee (EMC) and Board oversight and review.[6]

In recent years, the Department adopted a result- and priority-focused approach to G&C budget allocations. At the Board level, Outcome Project Groups (OPGs) submit G&C plans to their respective management boards, which then reviews the plans and makes recommendations to the Minister for the approval of agreements. Budgets are allocated in conformity with the agreements approved by the Minister. A mid-year review is also used to identify re-allocations across boards. Projects are selected on the basis of whether:

  • a legal obligation exists;
  • there is an obligation under a multilateral environmental convention that has been ratified by Canada;
  • the project advances Ministerial priorities;
  • funding is through an established funding program (either shared governance or within EC only); or
  • the project meets G&C approval form criteria (i.e. meets eligibility criteria under class; aligns with departmental priorities and results; presents a unique opportunity; leverages capacity).[7]

Two years ago, EMC established a G&C Management Committee to oversee the delivery of grants and contributions programs in the Department, including the classes. The committee includes an executive-level representative from each board, as well as one regional representative. A secretariat was also established, comprised of representatives from Corporate Finance and Intergovernmental and Stakeholder Relations (formerly Partnerships and Consultations). The committee focuses primarily on facilitating the identification of G&C priorities to be funded in a given year, although the official mandate of the committee is also to:

  • act as policy lead for G&Cs in the Department;
  • provide analysis to support Board decision-making;
  • conduct a mid-year review of G&Cs in the Department;
  • provide recommendations on signing authority levels for G&Cs; and
  • oversee ongoing management under the G&C framework.[8]

In terms of training and capacity-building for its managers and recipients, Environment Canada has had in place a manager's guide for the administration of Grants and Contributions, including the class G&Cs, since 1999. The guide is designed to assist program managers, financial officers and other interested parties in the proper and effective application of G&C programs and enhance accountability.[9]

2.2.1 Program Logic Model

The logic model for the class G&Cs presented on the following page is adapted from the Accountability, Risk and Audit Framework (ARAF) for the Departmental Class Grants and Contributions (March 2004). A logic model is a visual representation of a program or initiative that identifies the linkages between the initiative's activities and the achievement of its outcomes.

The objectives of the various initiatives funded through the class agreements, however, are sufficiently diverse that they cannot all be captured in a single unified logic model or set of outcomes. While loosely linked at the highest outcome level, the lower-order outcomes for which some attribution to the agreement could be made are far too numerous to accommodate in a logic model. Consequently, the logic model is presented only to provide a generic picture of the manner in which the class-related activities and outputs, as implemented through funded class agreements, contribute to the achievement of higher-order departmental outcomes.

Logic Model

3.0 EVALUATION DESIGN

3.1 Purpose and Scope

The evaluation covered the four class transfer payment authorities for the 2006-07 and 2007-08 fiscal years. Given that all four are managed and delivered in the same fashion using the same management, accountability, and financial reporting mechanisms, the four classes were treated as a single program when addressing issues concerning their relevance, cost-effectiveness, and design and delivery. Specifically, documentary evidence related to relevance and delivery applies equally to all four classes, while feedback from program managers and finance personnel, and to a lesser extent funding recipients, with broad experience in using all four classes is equally generic. Consequently, the only attempts to distinguish the four classes occurred in the context of the achievement of outputs and outcomes, addressed with evidence collected by means of the file review.

In terms of the timelines covered by the evaluation, the follow-up to the Audit and Evaluation of Class Grants and Contributions was completed in 2005-06 and addressed agreements for all four of the current classes up to that year. As such, there would have been little value in once again addressing these same agreements in the context of the current study. It should also be noted that agreements were tracked on a payment basis in 2005-06 and earlier, and on an agreement basis from 2006-07 onward. As such, inclusion of the 2005-06 agreements would have unduly complicated the file review analysis and so was of dubious advantage.

This evaluation assessed the relevance, success, cost-effectiveness/alternatives, and design and delivery of the class grants and contributions. The evaluation sought to determine whether these classes are:

  • aligned with and contributing to federal government priorities, and whether they address actual needs (relevance);
  • achieving or on track to achieving their intended outcomes (success);
  • using the most appropriate and efficient means to achieve their outcomes (cost-effectiveness/alternatives); and
  • designed and delivered in the best possible way (design and delivery).

To this end, the evaluation involved a variety of data-collection tools, including a document review, a review of files for class agreements and key informant interviews.

The unit of analysis for most lines of evidence was the individual agreements which fall under each class. Only a sub-sample of agreements from each class was reviewed, although efforts were made to stratify the sampling of agreements according to their year and value, as well as the extent to which they address similar objectives.

3.2 Evaluation Approach and Methodology

The following section provides a detailed description of the research activities undertaken for the project, as well as the identified research challenges/limitations.

3.2.1 Document Review

A primary source of information for the evaluation involved a review of existing documentation. The review included:

  • Performance documents (e.g., Accountability, Risk and Audit Framework [ARAF]);
  • Policy documents (Treasury Board submissions, Memoranda to Cabinet, departmental documents);
  • Existing performance reports (progress reports, financial reports);
  • Corporate planning documents (e.g., manager's guides, class terms and conditions); and
  • Audit and evaluation reports.

The document review responded to a number of evaluation issues related to the classes' ongoing relevance, overall success and cost-effectiveness.

3.2.2 File Review

In addition to a document review, a contractor was engaged to conduct a thorough review of a sample of agreement files. The review included a total of 104 files as follows:

  • 5 of 27 files for agreements funded through Grants to Support Environmental Research and Development;
  • 30 of 80 files for agreements funded through Contributions to Support Environmental Research and Development;
  • 45 of 608 files for agreements funded through Contributions to Support Regional Environmental and Sustainable Development Projects; and
  • 24 of 43 files for agreements funded through Contributions to Support International Commitments.[11]

While this approach did not ensure that all or even a majority of possible outcomes were considered during the review, it is important to note that the focus of the evaluation is at the class level and not on individual agreements. Thus, the proposed approach nonetheless provided adequate coverage to allow for an accurate assessment of the degree to which the classes are contributing to the achievement of departmental outcomes in general.

The file review included a review of such items as approval forms, contribution agreements, project activity reporting, financial files (e.g., requests for payment, cash flow statements, recipients accounting of expenditures), interview records, expense claims, audit results, agreement evaluation frameworks, and progress reports.

The vast majority of files reviewed (88%) contained a copy of the contribution agreement (CA) negotiated between the Department and the recipient and which serves as the basis for the transfer of funds (see Table 3.1). The Contribution Agreements typically reflect the Department's CA template.[12] The remaining 12% of files without a contribution agreement included only grants or contributions for which an agreement is not required, such as grants involving low risks or low materiality, assessed contributions,[13] or transfers to other federal government departments.[14]In addition to the CA, a large majority of files also contained records related to the financial aspects of the Agreement (88%). Financial information includes both project budgets that may have been included in the CA itself, as well as other financial records included in the files (such as invoices, cash flow statements, payment input/expenditure requests).

A significant portion of the files (70%) also included information on the proposal review process. For example, the Grant and Contribution Approval Form was widely used and some files also included the G&C Management Checklist. Far fewer of the class agreement files, 32%, included a copy of the project proposal.

Finally, 64% of the files included some form of activity or outcome reporting. For the minority of class agreement files that did not contain activity, output or outcome reporting, in roughly 15% of cases the agreement had no requirement for reporting (e.g., reporting is not required for Graduate Supplements or assessed contributions, such as memberships in international organizations). For the remaining 20% of files overall where there was a reasonable expectation of activity, output or outcome reporting, none was evident in the files.

Overall then, information submitted for the file review suggests that monitoring of class agreements is occurring fairly consistently. While there is some question as to whether all administrative information for the class contributions was available for review (see Section 3.2.4 on Challenges and Limitations for a detailed discussion), there would nevertheless appear to be some room for improving the monitoring of class agreements, particularly with regard to proposal reviews and output/outcome reporting.

Table 3.1 File Contents
Type of informationPer cent of files (n=104 )
* The remaining 12% of files without a contribution agreement included only grants or contributions for which an agreement is not required.
Contribution Agreement88*
Financial records88
Proposal review/approvals70
Final Report42
Progress reporting/activity reporting38
Proposal32
Correspondence18
Annual Report17
Meeting/committee agendas/minutes/proceedings14
Audit/evaluation report3
Other22

Prior to the full file review, a file review template was developed on the basis of a limited analysis of a sample of files. The template was used during the file review in order to capture file information in a consistent format, ensure that the sources of information gathered were documented, and to link the information to the specific evaluation questions and indicators. The file review addressed issues related to the consistency of contributions to departmental objectives, the achievement of outputs and outcomes, unintended impacts, and project costs.

A stratified random sampling approach was used to identify the sample of files for review. That is, files were randomly drawn from each of two fiscal year and three agreement size strata to ensure that the sample reflected as closely as possible the population characteristics on these variables. Table 3.2 demonstrates that the percentage distribution of agreements included in the file review sample is roughly parallel to that observed for the population of agreements overall. However, given the larger number of agreements signed using Contributions to Support Environmental and Sustainable Development Projects, it was necessary to under-sample agreements from this class and over-sample agreements negotiated using Contributions to Support Environmental Research and Development and Contributions to Support Canada's International Commitments.

Table 3.2 Distribution of File Review Population and Sample by Class Program
ProgramTotal number of agreementsSample
2006-072007-08
Total368390104
Grants to Support Environmental Research and Development18 (5%)9 (2%)5 (5%)
Contributions to Support Environmental Research and Development39 (11%)41 (11%)30 (29%)
Contributions to Support Environmental and Sustainable Development Projects297 (81%)311 (80%)45 (43%)
Contributions to Support Canada's International Commitments14 (4%)29 (7%)24 (23%)

Table 3.3 provides a profile of agreements included in the file review sample. Overall, the distribution of agreements sampled reflects a cross-section of class agreements according to region, year, dollar value and project scope. Although not an exact reflection of the distribution of all class agreements, the stratified sampling of agreement files ensured that, to the extent possible, the file sample reflected the population of all class agreements on key variables, and that it was appropriate for evaluation purposes.

Table 3.3 Characteristics of Agreements Reviewed
Agreement characteristicsPer cent of files (n=104)

Region

  • NCR
  • Ontario
  • Pacific and Yukon
  • Prairie and Northern
  • Atlantic
  • Quebec


32
27
16
16
9
2

Duration

  • Single year
  • Multi-year


75
25

Year agreement signed

  • Before 2006-07
  • 2006-07
  • 2007-08


19
40
42

EC agreement funding

  • $0-$24K
  • $25-$74K
  • $75-$100K
  • $101K+


21
33
19
27

Scope of operations

  • Regional
  • Provincial
  • National
  • International


53
12
11
25


3.2.3 Key informant Interviews

Key informant interviews were used to gather in-depth information on all the evaluation questions and to supplement information collected through the document and file review. A total of 41 key informant interviews were conducted with representatives from the following groups:

  • Departmental program managers with experience using class contributions (n=12) provided feedback on the need, consistency with priorities, outcome achievement, alternatives/cost-efficiency, design, accountabilities, performance measurement and best practices of the program;
  • Representatives of the Finance and Corporate Branch who oversee the financial review of agreements negotiated under the classes (n=4) provided feedback on issues related to alternatives/cost-efficiency, accountabilities, performance measurement, program management and best practices; and
  • Recipients of class grants or contributions funding (n=25, divided in rough proportion to the overall number of agreements supported by each class) contributed their perspectives on the need, outcome achievement, alternatives, design, accountabilities, performance measurement and best practices of the classes. Interviewees were selected from the sample of funded projects examined as part of the file review in order to better triangulate findings between the two sources of evidence.

Interviewees from the program and corporate finance groups were selected on the basis of their experience with larger numbers of agreements and/or agreements of a higher value. Interviews were conducted either in-person or over the telephone, depending on the availability and preference of the respondent, using semi-structured interview guides. Separate interview guides were developed for each respondent group to better reflect their background and experience with the classes and to ensure they were asked only those evaluation issues for which it was reasonable to assume they could provide an informed opinion.

3.2.4 Challenges and Limitations

Five key issues were considered in the design and implementation of the evaluation of the class grants and contributions.

1) An inability to link all class agreements to a small set of shared outcomes.

Instead of being linked together through common, shorter-term outcomes, agreements funded through the classes tend to share the particular outcomes of the departmental programs to which they are aligned. Departmental programs that use the class G&Cs are funded through a combination of operating and transfer funds and are designed to ensure that the resources, activities and outputs are well-aligned to the achievement of a small set of short-term outcomes. These short-term outcomes are in turn linked to higher-order, longer-term outcomes. It is only in the context of evaluating the programs to which the specific class agreements are aligned that meaningful conclusions as to the success of class agreements in achieving departmental results can be drawn.

Past attempts to study class grants and contributions as stand-alone entities[15] reveal that agreements funded through the classes are unified only at the highest outcome level. That is, agreements under the same class are linked only in so far as they address the same very broad theme (e.g., environmental research and development or international commitments). Evaluations rarely directly assess an initiative's progress toward such a high-level outcome, however, as it would be exceedingly difficult to uncover valid evidence of the achievement of these outcomes that could be reasonably attributed to the initiative under investigation. Evaluators must instead rely on evidence of the achievement of shorter-term, lower-level outcomes which can more easily be attributed to the initiative in question. Researchers will then deduce the initiative's effect on higher-order outcomes through the logical association of lower- to higher-order outcomes as depicted in a program logic model.

The difficulty in evaluating class grants and contributions arises when researchers attempt to link the various class agreements to a set of lower-order or shorter-term outcomes. Such a vast array of short-term outcomes exists for the myriad agreements funded through a single class that the outcomes would nearly equal the number of agreements themselves. Attempts to link the agreements at an intermediate outcome level may prove almost as difficult, as there were a total of 169 distinct medium-term results within the departmental results management structure to which agreements could be linked during the time period for the current evaluation.[16] Consequently, it is virtually impossible to link the agreements to a shared set of measurable outcomes, or to report on their achievement of these outcomes, in any coherent fashion.

Despite this challenge, attempts were made in the design and analysis of the current evaluation to address the degree to which projects have contributed to the achievement of departmental outcomes. The primary evidence used to address this issue comes from qualitative feedback from key informants and through file review evidence related to the degree to which agreement-level outcomes of funded projects had been achieved. It is important to note, however, that it was not possible to relate information from these lines of evidence to specific departmental outcomes. The evidence and ensuing discussion and analyses are limited in that they speak only in generic terms about the contribution of the classes to the achievement of departmental objectives, rather than linking the evidence to specific outcomes.

2) The absence of a centralized management information system.

The manner in which class agreements are tracked and maintained limits their accessibility and utility in developing sampling plans for evaluation purposes. Specifically, no central repository of class agreements is maintained, thus making it difficult to compile and assimilate files related to class contribution agreements. As such, there is some question as to whether all administrative information for the class contributions was available for review and thus, whether the findings from the file review provide a fully accurate assessment of the degree to which grants and contributions are managed appropriately and are achieving intended results.

For contribution programs with individual authorities, it is generally possible to access agreement files through a small number of program managers with, for example, no more than one contact per region for a given program. For class G&Cs and sub-classes that use the class terms and conditions, however, a large number of managers are able to access the classes for the purpose of establishing G&C agreements in support of their program areas. As such, the agreement forms and financial information for individual contribution agreements are maintained by seven Departmental Accounting Offices (six regional offices and the National Capital Region office), while other documentation in support of the agreement (e.g., project activity reporting, interview records, evaluation frameworks, progress reports) is maintained by the individual managers overseeing a specific agreement.

To address this issue, the evaluators contacted each of the Departmental Accounting Offices (DAOs) separately in order to request agreement files. The DAOs, in turn, were required to contact individual managers for additional requested file information. As such, the overall process of accessing agreement files required more time than anticipated. Furthermore, the degree to which multiple requests for file information and the decentralized records management of agreement files may have impacted the comprehensiveness of the agreement files available for review remains unclear.

3) A reliance on internal evidence.

Findings related to the classes' relevance and design and delivery are based exclusively on key informant interviews with departmental staff and program recipients. Given the scope of the evaluation, the extent to which these findings might have been corroborated through feedback from external stakeholders and partners was limited.

4) Limited information to assess cost-effectiveness.

The broad availability and use of the classes to support departmental programs meant that expenditures and human resources to support the administration of the classes are not captured separately. Although departmental key informants were able to provide qualitative feedback on the cost-effectiveness of the program, no administrative data exists to support these views.

5) Absence of a comparison group for projects funded under the classes.

The broad nature of the classes and consequent variety of projects funded through these class authorities meant that it was not possible to develop a comparison group against which to assess the success of class agreements relative to other projects that had not received class funding. In this manner then, it is not possible to determine conclusively whether progress toward the realization of departmental objectives as a result of funding third-party stakeholders using the classes is uniquely attributable to the classes.

4.0 FINDINGS

This chapter presents the detailed findings stemming from all lines of evidence used in the evaluation. Section 4.1 presents a discussion of findings related to the relevance of the classes. Section 4.2 explores the success of the classes, while the remaining sections of this chapter examine the cost-efficiency (Section 4.3) and design and delivery (Section 4.4) of the class grants and contributions.

A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings indicate that:

  • the intended outcomes or goals have been achieved or met--labelled as Achieved;
  • considerable progress has been made to meet the intended outcomes or goals, but attention is still needed--labelled as Progress Made, Attention Needed; or
  • little progress has been made to meet the intended outcome and attention is needed on a priority basis--labelled as Little Progress, Priority for Attention; and
  • the N/A symbol identifies items where a rating is not applicable.

A summary of ratings for the evaluation issues and questions is presented in Annex 4.

4.1 Relevance

Overall Findings

Environment Canada's class G&Cs continue to be an important means of supporting third parties in the pursuit of departmental priorities. Unlike conventional G&C programming, the broad scope of the classes provides managers across the Department access to important tools (grants and contributions) to further departmental objectives and to respond quickly to opportunities and changing priorities.

Class agreements tend to be well aligned to departmental priorities and are appropriately targeted to a variety of audiences that reflect the broad mandate of the Department. Numerous tools and processes have been established to ensure that agreements are well aligned to priorities, although there is a need to update some tools for managers to ensure they remain consistent with the Department's changing structure. There is no evidence to suggest that the classes are used to undertake activities for which programs are directly responsible or to avoid the development of unique G&C authorities.

4.1.1 Need for Class Grants and Contributions

Evaluation Issue 1: Is there a need for this class Grant and Contribution (G&C) to support third parties in the pursuit of departmental priorities related to (research and development/sustainable development/international commitments)? (Achieved)

  • To what extent does this class G&C overlap with or duplicate alternative funding sources to support these groups? (Achieved)
  • Why was a class contribution chosen over other funding alternatives?

Indicator(s)

  • Evidence of similar internal or external G&Cs
  • Demonstration of the utility/rationale for program/ funding mechanism
  • Views of program staff
  • Number of existing contribution programs and agreements that are subsumed within the classes

Methods

  • Document review
  • Interviews

In 2006, the report from the Blue Ribbon Panel on Grants and Contributions[17] concluded that federal grant and contribution programs represent one of the most important instruments through which the Government of Canada delivers on its responsibilities to Canadians. The panel notes that federal grant and contribution programs support a broad range of businesses, individuals, institutions, and communities across the country to carry out important work related to public policy. By enabling others, grants and contributions further federal priorities more efficiently and effectively by leveraging the skills, knowledge and commitment of various stakeholders working toward the achievement of similar objectives.

With very few exceptions, key informants interviewed for this evaluation feel there is an ongoing need for class G&Cs. At Environment Canada, these class transfer payment authorities are intended to provide funding to stakeholders and individuals who have a vested interest in the environment for the purpose of furthering the Department's environmental mandate and objectives, and supporting specific activities in partnering arrangements that extend departmental resources by leveraging funds from others and by supporting projects where the Department does not have the expertise or where others share jurisdiction.[18] As discussed in more detail in Section 4.2, the classes are widely regarded by key informants as successful in this regard.

Unlike conventional G&C programs, however, the Department's class grants and contributions were also designed to "facilitate managers' use of transfer payments as an effective and efficient means of engaging other Canadians to further environmental mandates."[19] The broad nature of the classes allows the Department to more easily align various small payments made under the classes to departmental results and priorities, and provides delegated managers better access to this important tool in support of program delivery. It is the efficient, flexible and accessible nature of the classes which underlies the views of many managers and finance personnel that there is an ongoing need for the class G&Cs.

In particular, program managers and finance personnel noted that the classes reduce administrative burden by avoiding the need to develop new terms and conditions for every transfer payment requirement. A review of financial administrative files for the classes suggests that classes have done a great deal to relieve this administrative burden, as the class terms and conditions were used to support nine sub-classes in 2006-07 and 12 sub-classes in 2007-08.

Even more importantly, however, all groups are of the view that the flexibility conferred by the classes is essential to quickly adjust funding requirements to respond in a timely fashion to changing departmental priorities and engage in projects that might not fall within the purview of more narrowly focused G&C programs. As well, one finance representative notes that the classes make "...it easier for the manager to incorporate G&Cs into their program design and delivery." In this manner then, the classes are instrumental in ensuring that all departmental managers have access to G&C tools for their program delivery and in closing funding gaps to support the work of stakeholder groups in advancing departmental priorities. This flexibility comes at a price, however, as some finance respondents remarked that it also makes it more difficult to measure and report on the results of classes (see Section 4.3.2.1 for a more detailed discussion).

Program managers and finance personnel generally feel that all four classes are important to support the work of the Department,[20]and overlap or duplication with other funding sources was not felt to be an issue for any respondent group. In the view of one manager, however, without a centralized information system accessible to managers, it may be impossible to completely avoid overlap.

Summary: Environment Canada's class G&Cs continue to be an important means of supporting third parties in the pursuit of departmental priorities. Unlike conventional G&C programming, the broad scope of the classes provides managers across the Department access to important tools (grants and contributions) to further departmental objectives and to respond quickly to opportunities and changing priorities.

4.1.2 Alignment with Departmental Priorities

Evaluation Issue 2: To what extent do the class G&Cs support projects that are consistent with departmental priorities? (Achieved)

  • Are the classes used to support agreements aligned directly with program activities?

Indicator(s)

  • Consistency of agreement with program and departmental objectives
  • Existence of gaps in the coverage of class objectives
  • Reach is analyzed and targeted
  • Both the reach and activities are connected to environmental requirements
  • Views of program staff
  • Proportion of agreements and contribution programs that align directly with program activities
  • Proportion of agreements to be evaluated as part of programs to which they align.

Methods

  • Document review
  • File review
  • Interviews

A review of documentation related to the administration of Environment Canada's class G&Cs programming demonstrates that a variety of mechanisms have been instituted to ensure alignment with departmental priorities, including:

  • Separate terms and conditions for each of the four class G&Cs which outline their specific objectives and the departmental key strategic outcomes (long-term key results) that each class is to achieve, as well as presenting reach (eligible recipients).[21]
  • Financial coding, which "is an essential element for linking the individual contribution agreement generated by the manager to the higher level results for the purposes of performance measurement, evaluation and audit."[22]
  • An annual G&Cs project review and approval process involving OPGs, Boards, Finance and the Minister, which ensures decisions with respect to planned spending on G&Cs are based on departmental strategic priorities.[23]

Although the existing terms and conditions for the class authorities outline the departmental strategic outcomes based on the previous departmental business line matrix management accountability structure, not the present results-based management structure (implemented in fiscal year 2005-06), other processes, such as the approval and financial review processes, provided context related to the alignment of class G&Cs to the departmental results-based management structure which existed during the timeframe in question.[24]

Program managers and finance personnel agree that the class G&Cs support projects that are consistent with departmental priorities. Several respondents from each group remarked that changes to the G&C allocation and approvals processes introduced over the last several years, and in particular the level of internal scrutiny at the OPG, Board, and Ministerial levels, ensures that all G&C agreements are directly aligned to departmental results and priorities. Any agreements not directly aligned with these priorities have been phased out since these changes have been introduced.

These views are consistent with findings from the review of agreement files, which found that contribution agreements and G&C approval forms for all files reviewed clearly demonstrate the link between the project and departmental priorities. Over nine in ten files linked the class project to one of the strategic outcomes associated with Ecosystem Sustainability (62%), Weather and Environmental Services (25%) or Environmental Protection (6%), while the remaining agreements were linked to other departmental outcomes.[25]Similarly, documentation relating to the assessment of project proposals for funding was provided for 70% of files and of these, the majority (86%) were determined to be moderate to very clear in their assessment of projects as being aligned with funding program criteria and priority areas.[26] Most agreements (94%) were also judged to include a moderate to very detailed discussion of the rationale for the project activity and a link to environmental requirements.

Reflecting the breadth of departmental priorities, the file review also demonstrates that the work undertaken through the class agreement projects targeted a wide range of relevant departmental stakeholders. Researchers and scientists are the primary target audience for 63% of the class agreement projects that were reviewed, followed by First Nations groups (27%), community leaders (18%) and municipal staff or elected officials (16%). One in ten projects include provincial staff or elected officials (13%), the general public (11%), or other countries as target audiences (10%). Other project target audiences consist of NGOs, business or the private sector, the federal government or real estate developers.

Table 4.1 Audiences Targeted by Class-Funded Project Activities
Agreement reachPercent of files (n=104)*
* Totals sum to more than 100%, as multiple target audiences were possible.
Researchers/scientists63
First Nations27
Community leaders18
Municipal staff/elected officials16
Provincial staff/elected officials13
General public11
Other countries10
NGOs8
Business/private sector8
Federal government3
Real estate/developers2
Other13

While it is important that projects funded through the classes are aligned with departmental priorities, it is equally important that class grants and contributions are not used to carry out the activities of departmental programs (i.e., activities which, if carried out by a third party, would need to be contracted). Departmental program managers were unequivocal in stating that the class G&Cs agreements are not used to undertake activities for which programs are directly responsible. Many managers and finance respondents variously pointed out that: these agreements are used to support activities that the Department is not able to undertake itself; the question of when to use a contribution agreement versus a contract has been clarified over the past two to three years; using the class G&Cs to support agreements that are directly aligned with program activities is a contravention of the intent of the G&C Vote; and control mechanisms exist within the Department to ensure that G&C agreements (including those negotiated under the classes) do not align directly with program activities but support their work. The role of financial personnel in this context is to ensure that the G&C agreements, including those supported through the classes, meet all of the requirements of the Transfer Payment Policy, including assurance that the activities undertaken under the agreement do not align directly with departmental activities.

A review of administrative databases for the classes also failed to yield evidence that the classes are being used to avoid the development of unique G&C authorities. Unique authorities are developed in cases where there is significant amount of G&C funding activity aligned with a single departmental program area. The unique authority is established to improve accountability and reporting related to the administration and performance of the group of similarly aligned G&C agreements. Sub-class G&Cs are one example of a unique authority which is developed using the class terms and conditions (thus reducing administrative burden) such that spending through the sub-class program is tracked separately and program accountabilities rest with the manager responsible for the program area to which the unique sub-class G&C program is aligned. As mentioned previously in Section 4.1.1, a review of administrative files for the class G&Cs shows that the classes were used to support nine sub-classes in 2006-07 and 12 sub-classes in 2007-08. The relative frequency with which the classes are used to support these unique sub-class program authorities suggests that the classes are not being used to avoid the development of unique authorities. Further, these figures suggest that the classes may encourage the development of unique authorities by reducing the administrative burden associated with the development of new terms and conditions.

Summary: Class agreements tend to be well aligned to departmental priorities and are appropriately targeted to a variety of audiences that reflect the broad mandate of the Department. Numerous tools and processes have been established to ensure that agreements are well aligned to priorities, although there is a need to update some tools for managers to ensure they remain consistent with the Department's changing structure. There is no evidence to suggest that the classes are used to undertake activities for which programs are directly responsible or to avoid the development of unique G&C authorities.

4.2 Success

Overall Findings

The intended results of agreements funded through the classes have for the most part been achieved, although this was not always evident for ongoing projects, in cases where funding delays negatively affected the achievement of project results or when other barriers (e.g., technological issues, environmental circumstances) were present. Performance-measurement information was most likely to be output-based, although for many agreements (e.g., with longer-term outcomes, outcome information not stipulated/appropriate, or those that present measurement challenges) information related to outputs is all that could reasonably be expected.

The classes were instrumental in the achievement of project results in a large majority of cases. These class authorities were often either the only funding source, a catalyst to leverage resources, or sufficiently flexible to allow the project to proceed as designed and/or be adjusted as circumstances warranted.

The classes are widely regarded as contributing to the achievement of departmental objectives, most notably because the classes leverage resources (e.g., partners, expertise) and provide the flexibility to support new priorities, a range of activities and can be strategic in the direction of funding. The impact of the classes on the achievement of departmental priorities has been dampened somewhat by a long and complex agreement approvals process.

There are few documented instances of unintended outcomes. Key informants were most likely to provide anecdotal evidence of positive unintended impacts related to exceeding intended project results. The one negative impact raised by all respondent groups concerned the potential for recipient organizations to develop expectations of ongoing funding from the class authorities and an over-reliance on class funding.

4.2.1 Achievement of Project Outcomes

Evaluation Issue 3: To what extent have the intended outcomes of funded projects been achieved as a result of the class G&Cs? (Achieved)

  • To what degree have funded projects contributed to the achievement of departmental objectives? (Progress made, attention required)

Indicator(s)

  • Evidence of intended output and outcome achievement
  • View of program staff and recipients

Methods

  • Document review
  • File review
  • Interviews

Previous audit and evaluation reports identified a need to make more visible the linkage between grants and contributions and the realization of program and business line goals and objectives (now Board results and project outcomes). The Follow-up to the Audit/Evaluation of Class Grants and Contributions[27] concluded that although an ARAF was completed for class G&Cs, the reporting obligations on specific class contributions remained imprecise. Specifically, not all G&C programs were capturing results-based data (in a systematic manner) and reporting these to departmental senior management.

A number of mechanisms have been established to ensure that grants and contributions programs, including the class G&Cs, be oriented toward the achievement of outcomes. An Accountability, Risk and Audit Framework (ARAF) sets out the results expected from class G&Cs in the Department and requires that each grant and contribution agreement identify outcomes/results and, in the case of contributions, expected deliverables. Payments are to be made based on the achievement of performance objectives (i.e., deliverables), as outlined in each contribution agreement (see Section 4.4 below for a more in-depth discussion). Furthermore, departments are expected to provide in their Departmental Performance Reports evidence of results achieved for each transfer payment program with transfers in excess of $5 million.[28]

Funding recipients and program personnel were virtually unanimous that the intended outcomes of class-funded projects had by and large been achieved. More than half of the funding recipients interviewed report that the outcomes of their projects had been achieved fully, while the remaining respondents note either that a large majority of outcomes were achieved or that the project exceeded expectations as a result of additional resources and input contributed by other project partners. In describing the achievement of outcomes, project proponents interviewed were most likely to refer to outcomes or a combination of outcomes and outputs, while relatively few focus solely on project outputs.

Findings from the review of agreement files are consistent with the feedback received from key informants, and demonstrate that agreement files are somewhat more likely to uncover evidence related to the achievement of project outputs than outcomes. Performance reporting for class agreements typically took the form of progress or final project reporting and two-thirds (64%) of the files contained some form of this type of reporting.[29] Of these agreement files, relatively few (15%) included a report in the file confirming that project outcomes were evaluated by an external consultant (2%) or that steps were taken to evaluate project activities, such as the results of project workshops or conferences (13%).

Over two-thirds demonstrated moderate to very clear evidence of outputs (69%),[30] compared to four in ten agreement files that included a moderate to very clear description of project outcomes achieved (39%).[31] Material related to outcomes (as well as outputs and evaluation planning and final reporting) is more prevalent among project agreements funded by Contributions to Support Environmental and Sustainable Development.

Most managers feel project planning (e.g., the articulation of project objectives and indicators) and project oversight by Environment Canada staff does a great deal to contribute to the achievement of project outcomes. Furthermore, several respondents note that their familiarity with performance measurement and reporting related to class-funded agreements allows them to conclude that project outcomes are being achieved. One manager felt that for certain types of projects where deliverables are less clear, such as consultations, it is more difficult to be certain that project objectives have been achieved.

4.2.1.1 Challenges in Achieving Project Objectives

The majority of managers volunteered that not all projects meet all of their stated objectives. Similarly, roughly one-third of recipients reported that not all outcomes related to the projects about which they were being interviewed had been achieved. Rather than being indicative of a failure to meet project milestones, however, these findings are instead reflective of recipients' broader interpretation of project outcomes, as nearly all recipients explained that the overall project work was ongoing and targeting longer-term outcomes and objectives. Others pointed out that funding delays or difficulties with project partners also negatively impacted the achievement of project outcomes in some cases. Several managers noted that in these cases, they will take steps to enhance the achievement of objectives through the provision of additional support or the adjustment of funding schedules. One recipient also pointed out that their project evaluations identify lessons learned from less successful projects and that these are taken into account to adjust any future projects.

Similarly, the file review found that for a small number of agreements there was moderate to poor alignment between the project design and its implementation. While often not explicitly documented or detailed, the reasons for the divergence between design and implementation were found to be associated with a variety of challenges such as: complexity of the project; technological issues; delays in contract signature/receipt of funding; overly ambitious objectives/inadequate timeframe; weather/seasonal/ environmental circumstances; and data collection challenges.

It is also important to note that the absence of outcome reporting in project files does not necessarily imply that performance measurement is not being conducted; rather, some projects that are output-based may be capturing activity information (as opposed to outcome data). As well, for some projects, outcome data may be neither a required nor an appropriate stipulation of the agreement (e.g. , when project is purely output-based, such as basic research). Finally, projects may be funded to achieve longer-term outcomes (e.g., enhanced prediction of weather events) or outcomes that present measurement challenges (e.g., enhanced prediction of weather events to reduce weather-related hazards), thus making an assessment of ultimate outcomes highly difficult.

Summary: The intended results of agreements funded through the classes have for the most part been achieved, although this was not always evident for ongoing projects, in cases where funding delays negatively affected the achievement of project results or when other barriers (e.g., technological issues, environmental circumstances) were present. Performance-measurement information was most likely to be output-based, although for many agreements (e.g., with longer-term outcomes, outcome information not stipulated/appropriate or those that present measurement challenges) information related to outputs is all that could reasonably be expected.

4.2.1.2 Importance of Classes to the Achievement of Project Outcomes

Several managers and a large majority of funding recipients indicate that project results either would not have been achieved at all or would have been seriously diminished without funding through the class grants and contributions. Among managers, the key reason for feeling class funding is instrumental to project success is that it allows recipients to leverage these funds to secure further resources from external partners. Some respondents note that the Environment Canada funding is often the catalyst to attract stakeholder support for these projects, and that without first securing federal funding the projects would not have materialized at all, as there would have been nothing with which to leverage other funding.

Feedback from recipients touched on many of the same issues raised by managers. Roughly four in five funding recipients felt that not all of their project outcomes would have been achieved without funding from the class grants and contributions. Specifically, nearly half of all recipients reported that their project outcomes would not have been achieved at all without this funding and approximately one-third felt their projects might have proceeded but not achieved all project outcomes. These recipients' primary reasons for feeling that not all project outcomes would have been achieved were that the class funding:

  • was the sole or one of a very limited number of funding opportunities for their types of projects;
  • was necessary to leverage partnerships or encourage collaboration with other stakeholders;
  • addressed a critical aspect of their projects, such as implementing actions in response to issues identified through funded research or attracting the right types of partners; and
  • afforded the flexibility necessary for their project to proceed as designed and to adjust as warranted by the circumstances.

Some recipients also note that the class funding was used to support Canada's obligations under international or multi-jurisdictional agreements. Although these agreements would likely have continued to varying degrees without Canada's participation, the class funding was viewed as essential in order to demonstrate Canada's commitment to the issues addressed through these agreements, to fulfil its legal obligations, and/or to avoid embarrassment and maintain its reputation.

Relatively few funding recipients indicated that project results would have been realized regardless of class funding, but these projects represented either students in receipt of grants or, as discussed above, funding to fulfil Canada's obligations under international or multi-jurisdictional agreements.

Summary: The classes were instrumental in the achievement of project results in a large majority of cases. These class authorities were often either the only funding source, a catalyst to leverage resources, or sufficiently flexible to allow the project to proceed as designed and/or be adjusted as circumstances warranted.

4.2.2 Achievement of Departmental Objectives

Most managers and finance personnel are strongly of the view that the class G&Cs have contributed to the achievement of departmental objectives. Those few managers who expressed some uncertainty in this regard noted only that some projects were at too early a stage to demonstrate their contribution to the achievement of objectives or "too grassroots" to confidently link them to higher-order departmental outcomes.

Managers tend to point to the fact that class grants and contributions are an important vehicle for achieving the Department's policy goals, by:

  • supporting the efforts of partners and other jurisdictions;
  • supporting a significant portion of some departmental program activities (i.e., through sub-classes);
  • allowing the Department to harness capacities that do not exist internally, while influencing the direction of funded projects in order to achieve objectives; and
  • contributing to the work of departmental programs, such as through access to existing networks, research that provides insights into stakeholder groups or promotion of the program's position.

Both finance personnel and managers also point out that a particular advantage of the classes over more conventional G&C programming, and which is essential to the implementation of some projects related to departmental priorities, is their flexibility. It is this characteristic of the classes that allows the Department to:

  • quickly and efficiently shift the use of the classes to support new priorities as these emerge;
  • support a range of activities in a variety of priority areas; and
  • direct funding to where it will achieve the best results.

In the view of several managers, however, the flexibility of the classes and the associated benefits in achieving project and departmental outcomes may be threatened as a consequence of the complex approvals process involving senior managers, the Priority Management Boards, the Minister's office, and the Finance and Corporate Branch. Within the timeframe for this evaluation, the complexity of these approvals has led to delays in issuing funding to recipients. Although one respondent noted that the effect of these delays can be mitigated to some extent by modifying funding schedules and work plans, another pointed out that in some cases, the delays were such that there was too little time remaining in the fiscal year to achieve project objectives. In addition to these project-level impacts, at the class level these funding delays meant there was insufficient time to identify areas where additional resources were still available, to reach agreement on how the funds are to be reallocated, and to move these funds to other classes as required.

Summary: The classes are widely regarded as contributing to the achievement of departmental objectives, most notably because the classes leverage resources (e.g., partners, expertise) and provide the flexibility to support new priorities, a range of activities and can be strategic in the direction of funding. The impact of the classes on the achievement of departmental priorities has been dampened somewhat by a long and complex agreement approvals process.

4.2.3 Unintended Impacts

Evaluation Issue 4: Have there been any unintended (positive or negative) outcomes at the level of the class G&Cs? At the level of the individual agreements? (N/A)

  • Were any actions taken as a result of these?

Indicator(s)

  • Presence/absence of unintended outcomes
  • Documented management actions and/or lessons learned from unintended outcomes
  • Views of program staff and recipients

Methods

  • Document review
  • File review
  • Interviews

In nearly all agreement files reviewed (98%), unintended outcomes of the project were not acknowledged or identified. Included in this result are projects with no documentation on outcomes; however, after adjusting for the presence of outcome-related documentation, those with a moderate to very clear description of unintended outcomes are still the minority. While this result appears dramatic, reporting of unintended outcomes is typically not considered or formally requested of projects. Even for projects for which reporting templates were available and completed, reporting on unintended or unanticipated outcomes was not formally requested.

Roughly half of the program managers and two-thirds of recipients interviewed identified unintended positive outcomes resulting from funded projects and/or the classes themselves. A number of respondents from both groups commented that the classes exceeded expectations with regard to project partnerships, whether by attracting more project partners than anticipated, improving working relationships, laying the groundwork for possible future collaborations, or enhanced international cooperation.

These respondents also pointed to a number of other unforeseen benefits for their projects or organizations. While two managers expressed surprise at how quickly the projects with which they were involved had moved forward as a result of the class funding, several recipients also pointed to benefits such as recognition from the international community on the effectiveness of their management model, the publication of an extra journal article, enhanced expertise in one particular issue area, better-than-expected progress in working with municipalities, and longer-term employment and recruitment into the research workforce in communities. One recipient also noted that some of their staff has been recruited to work for Environment Canada, thus having a positive impact on the Department.

The one negative impact of the classes that was raised by all respondent groups was the potential for recipient organizations to develop expectations of ongoing funding from the class authorities and an over-reliance on class funding. To allay this sort of misunderstanding, one manager pointed out that those who have had to deal with complaints from organizations now take the time to explain up front that funding is not guaranteed. Another also commented that delays and/or cessations of funding have put an end to these expectations and the dependency of organizations on class funding.

A few other unintended negative impacts were also identified by:

  • one manager, who noted that some have the mistaken impression that the classes constitute distinct programs, rather than tools to support programs;
  • one finance officer, who felt that inexperienced managers occasionally try to exert too much influence on contribution agreements once established, although the creation of a risk-tolerance strategy (required under the new Transfer Payment Policy[32]) reduces the chances of this occurring; and
  • one recipient, who indicated funding delays had led to a temporary lay-off of some staff.

Summary: There are few documented instances of unintended outcomes. Key informants were most likely to provide anecdotal evidence of positive unintended impacts related to exceeding intended project results. The one negative impact raised by all respondent groups concerned the potential for recipient organizations to develop expectations of ongoing funding from the class authorities and an over-reliance on class funding.

4.3 Cost-Effectiveness

Overall Findings

Agreements negotiated through the class G&Cs are well-aligned with departmental priorities and adequate tools and processes are in place to ensure that this is the case. Qualitative and file review information suggests that the classes have been quite successful in leveraging funds from recipients.

Administrative costs and overall project costs were not available to directly assess the costs of delivering the classes. Nonetheless, the class G&Cs are generally felt to be the most appropriate and/or cost-efficient means to support stakeholders in the pursuit of departmental priorities when compared to any of the other tools that exist to support third parties (e.g., contracts, unique G&C authorities, partnership agreements). The flexibility of the class G&Cs to respond to opportunities, adjust to changing priorities, and re-allocate funding was the primary reason most respondents felt this way, as these characteristics of the classes allow the Department to reduce administrative burden, leverage more funding and avoid lapsing funds.

Suggestions to improve the efficiency of the classes related primarily to the implementation of a more balanced risk-mitigation approach to their delivery, including more timely approvals and funding, delegating some authorities for approvals, clarifying roles and responsibilities for their delivery (and so avoiding duplication), balancing flexibility and accountability, and engaging in multi-year agreements.

Overall, managers and finance personnel tend to identify more advantages than disadvantages of the class approach to grants and contributions, and are much more likely to be of the view that the advantages outweigh the disadvantages. Both the advantages and disadvantages of the classes were typically felt to involve their administration (e.g., nationally consistent approach and oversight by Finance versus resources required to oversee a broadly accessible program) and design (e.g., flexibility to respond to priorities and breadth of scope/accessibility versusdifficulty in aggregating agreement outcomes and short timelines).

A minority of respondents were of the view that relative to unique G&C programs, evaluation, accountability and reporting are easier under the classes as a result of the nationally consistent approach and expertise that has evolved for their management. More frequently, however, the broad nature of the classes was felt to diminish the ability to evaluate and report on them (i.e., outcomes are too disparate to be sensibly aggregated) or to assign clear accountability for the classes. The absence of a centralized information management system for the classes further exacerbates the difficulties in evaluating the classes and measuring their performance.

4.3.1 Value for Federal Dollars

Evaluation Issue 5: Have the class G&Cs provided value for federal dollars spent? (Achieved)

Indicator(s)

  • Analysis of costs of classes compared with achievement of outcomes
  • Selection process for areas of importance are applied

Methods

  • Document review
  • File review
  • Analysis completed for outcomes question (#3)

Environment Canada's spending on class grants and contributions represents only a small proportion of overall departmental spending, but a large proportion of the overall departmental G&C spending. Total actual departmental spending for the two years examined in this evaluation was $868.4 million in 2006-07[33] and $997 million in 2007-08.[34] Of these funds, spending through the class G&Cs accounted for 2.2% of total spending in 2006-07 ($19 million) and 2.6% of total spending ($26 million)[35] in 2007-08. Although they account for a relatively small proportion of overall departmental spending, the class G&Cs represent a large share of the Department's overall grants and contributions spending, accounting for 40% and 24% of overall G&C spending in 2006-07 and 2007-08, respectively. As discussed in Section 2.1.1, however, various difficulties related to tracking and updating class agreement information and reconciling information between the Department's financial reporting and database systems meant that it was not possible to arrive at a definitive profile of the classes, both in terms of the number and overall value of class agreements funded during the 2006-07 to 2007-08 timeframe.

The issue of whether the Department receives value for its class G&Cs spending concerns both the question of whether the Department receives a good return on its class G&Cs investment, and whether these investments were the appropriate ones to make in light of what the Department is trying to achieve.

On the question of whether the class G&Cs investment is appropriately targeted, the evaluation yielded a great deal of evidence to suggest that the agreements supported through the classes are well-aligned with departmental priorities. The file review found that all agreement files identified links to the Department's strategic outcomes, and numerous key informants spoke at length about the departmental grants and contributions approvals process, which ensures that G&C funding is strategically focussed toward priority areas (see Section 4.1.2 for a more detailed discussion). A number of tools are also available to managers to ensure that agreements are well aligned with departmental priorities. The file review shows that in a large majority of cases these tools, such as contribution agreement templates (100% of cases where a contribution agreement was required) and proposal reviews and approvals (70%), have been employed in the negotiation of class grants and contributions agreements.

Evidence also exists to explore the issue of whether the Department receives a good return on its class G&Cs investment. The file review found that one-third (34%) of all agreements reviewed identify partners. Most (57%) of these agreements were formal partnerships (e.g., listed in the contribution agreement itself) and the majority (63%) identified three or more partners. Partners were most likely to be from government sectors (municipal, provincial, federal) (32%), educational institutions (29%) or First Nations (27%), followed by private-sector organizations (25%), community-based organizations (21%), professional associations (18%) and other countries (18%). The partnerships identified by means of the file review most often involved funding (69%), but many also reported other "in-kind" investments from partners, including research (31%), advisory roles (26%) and service delivery (14%). For those partnerships providing funding, the amounts range from less than $50,000 (25%), and $50,000 to $150,000 (29%), to over $150,000 (46%).

The file review also yielded direct evidence of leveraged funding. While information related to the full value of the projects (i.e., sum of EC funding and funding leveraged from other partners) is not tracked centrally for all active agreements, this information was nonetheless captured through the contribution agreements for 91 of 104 agreement files reviewed. For these 91 agreements, Environment Canada funding totalled $14,207,184, with average EC funding of $156,123 per project. The total value of these 91 projects was $92,676,880, with an average total project value of $1,018,427. A comparison of EC funding to total project value demonstrates that every dollar of Environment Canada funding was matched by $5.50 of funding from non-EC project partners.[36]

Other lines of evidence also contributed indirect evidence of leveraging. One of the primary purposes of the class grants and contributions is to "solicit and encourage the participation of a broad spectrum of both domestic and international recipients to support EC's objectives [which] provides an effective mechanism to lever the departmental funds and greatly enhances...program delivery capability."[37] Funded recipients are encouraged to solicit funds from other sources (either financial or "in-kind"),[38] and to engage partners to pursue activities for which the Department lacks the necessary expertise.

Results of the key informant interviews and document review support the notion that the classes have been effective in this regard. Many key informants pointed out that class agreements allow program managers to access capacity and expertise that does not exist within the Department, such that all class agreements can be argued to have used departmental funds to leverage recipient resources. Secondary evidence also provides examples of class G&Cs funding that has been well leveraged, although there has been no attempt to quantify these expenditures for classes in their entirety. Specifically, an economic impact analysis of the Atlantic Coastal Action Program (ACAP)[39] found that "it would have cost Environment Canada 10 times their current ACAP program expenditures if the Department had used direct delivery to accomplish what the 14 ACAP organizations accomplished during 2001-02 to 2007-08."[40]

Summary: Agreements negotiated through the class G&Cs are well-aligned with departmental priorities and adequate tools and processes are in place to ensure that this is the case. Qualitative and file review information suggests that the classes have been quite successful in leveraging funds from recipients.

4.3.2 Efficiency of Delivery

Evaluation Issue 6: Are there alternative ways of supporting third parties in the pursuit of departmental priorities? (Progress made, attention needed)

  • If the class G&Cs continues, how could its efficiency be improved?

Indicator(s)

  • Analysis of delivery options/opportunities
  • Views of Finance and program staff

Methods

  • Document review
  • Interviews

The broad nature of the class G&Cs and their frequent use in the context of a much wider array of program activities means that it is not possible to capture administrative costs associated with the delivery of the class G&Cs.

Nonetheless, most managers and finance personnel feel that the class G&Cs approach is the most appropriate approach to support individuals, organizations and other levels of government in the pursuit of departmental priorities. While several managers and finance personnel noted that one alternative would be to develop unique G&C authorities, this option was also generally dismissed as either not cost-effective, unable to address gaps in G&C support to different program areas or not sufficiently flexible to allow the Department to re-allocate G&C funding. Some finance personnel noted that any alternative G&C approach would need to reduce administrative burden (i.e., avoid going to Treasury Board for every new G&C requirement) and maintain flexibility, while also being amenable to evaluation.

Respondents from both groups also frequently mentioned other means of supporting the work of external parties, such as through contracts (where a direct service or product is being delivered), collaborative/partnership agreements, and joint projects or in-kind support, but it was generally agreed that these would not be appropriate for the specific types of projects supported by the classes. As one manager pointed out, different approaches to support third parties "... intertwine as a menu of delivery options to achieve our priorities." Furthermore, it was often pointed out that as with unique G&C authorities, these other tools do not provide the same flexibility to respond to opportunities and adjust to changing departmental priorities, and are not as cost-effective (i.e., if the limited flexibility means funds are lapsed or if money is not leveraged) as the classes.

In terms of efficiencies at the project level, almost all recipients felt that the approach they had taken to achieve their project's objectives was the right approach. Nonetheless, many felt that improvements could be made to ameliorate their projects, most notably by increasing partnerships and collaborative arrangements with other relevant stakeholders (e.g., federal, provincial/territorial, private industry, First Nations).

4.3.2.1 Suggestions to Improve the Efficiency of Class G&Cs

It is widely recognized that there is a need to improve the efficiency with which the Department and all federal agencies manage and deliver grants and contributions programs. In its report From Red Tape to Clear Results: The Report of the Blue Ribbon Panel on Grants and Contributions, the panel not only concluded that there is a need to change the way the federal government "understands, designs, manages and accounts for its grant and contribution programs," but that it should also be possible "to simplify administration while strengthening accountability"[41] of these programs. The overall report also features 32 recommendations to cut red tape and strengthen accountability.

Key informants were similarly inclined to suggest ways to improve the efficiency of the classes, as a variety of suggestions were forwarded by all respondent groups, including:

  • More timely approvals and funding. Most managers and recipients, as well as some finance personnel, noted that there was a need to improve the timing of project approvals and funding. In particular, managers and finance personnel felt that the current processes demonstrate a low tolerance for risk, with the same time and rigour spent on reviewing small and/or renewed agreements as is spent reviewing large and/or newer agreements.
  • Delegated authorities. Some respondents from all groups suggested process efficiencies would be realized by returning some delegated authorities to local managers while ensuring accountability frameworks are in place. For example, one manager felt this would allow managers to exercise some discretion in terms of re-allocation of G&C funds or changing the terms of specific agreements. A finance respondent also noted that inconsistencies in delegation of authority between different G&C programs (e.g., one program might delegate authority up to $100,000 while another might delegate authority up to $2 million) leads to a "convoluted" approval process through which the program manager must manoeuvre.
  • Clearer accountabilities. Managers and recipients felt that efficiencies could be realized with greater clarity around the roles and responsibilities of the various players involved in the management and delivery of the classes. A couple of managers suggested there may be duplication of process as both managers and finance personnel examine deliverables and expenditures, while one recipient felt there was no clear business planning process that addresses the roles and responsibilities of the players involved. Another recipient suggested the creation of a central G&C Secretariat that would be responsible for the preparation of all G&C agreements within the Department, which may be similar to a model adopted by NRCan for the support of their class G&C program.[42]
  • Striking a balance between flexibility and accountability. Managers and finance personnel felt the class G&Cs need to balance flexibility with the ability to account for results. One manager suggested that if too narrowly defined, the terms and conditions of class G&Cs would limit the Department's ability to shift funds to respond to priorities and opportunities. Two finance representatives were of the view that efficiencies might be realized if the classes were adjusted to maintain their flexibility while focusing on outcomes and results that can be evaluated and linked directly to departmental results (as per the PAA).
  • Standard tools and processes. A number of finance personnel and some recipients suggested a variety of adjustments to tools and processes that might improve the efficiency with which the classes are administered. These include: clear guidelines on when to use various tools (e.g., G&Cs, contracts, MOUs, etc.) given project objectives;[43] improved guidelines and templates for the management of agreements, which outline the requirements of agreements; a management information system to track all G&C agreements; and clearer communications (e.g., through information sessions) related to the intent of the classes, the Department's strategic priorities, and the level and availability of funding prior to negotiating agreements.
  • Multi-year agreements. Multi-year agreements are contributions that extend beyond a single fiscal year timeframe, thus commiting the Department to providing contribution funding to a given recipient over multiple fiscal years. One finance officer felt that there is a need to re-examine multi-year agreements and to look for alternative ways to support these types of relationships. They point out that the present approvals process does not lend itself to ensuring these types of agreements are aligned with departmental priorities (i.e., the priorities may change), and so these types of agreements are less likely to be supported. One recipient also expressed frustration at not being able to enter into a multi-year agreement for their ongoing work.
  • Adopt a program focus. One manager felt the use of class G&Cs could be made more efficient if G&Cs, including those supported through the classes, were examined within the overall context of the programs with which they align. In this way, when planning their program delivery, managers would assess whether they have the right mix of class and/or unique G&Cs, O&M and salary funding or even whether G&Cs (class or otherwise) are an effective tool for a given program.

Summary: Administrative costs and overall projects costs were not available to directly assess the costs of delivering the classes. Nonetheless, the class G&Cs are generally felt to be the most cost-efficient means to support stakeholders in the pursuit of departmental priorities. The flexibility of the class G&Cs to respond to opportunities, adjust to changing priorities, and ensure that G&C funding is not lapsed was the primary reason most respondents felt this way.

Suggestions to improve the efficiency of the classes related primarily to the implementation of a more balanced risk-mitigation approach to their delivery, including more timely approvals and funding, delegating some authorities for approvals, clarifying roles and responsibilities for their delivery (and so avoiding duplication), balancing flexibility and accountability, and engaging in multi-year agreements.

4.3.3 Advantages and Disadvantages of a Class Approach

Evaluation Issue 7: What are the advantages and disadvantages of using a class approach to support third parties in the pursuit of departmental priorities? (N/A)

  • Do the advantages outweigh the disadvantages?
  • What are the implications for evaluation, accountability and reporting? (Little progress, priority for attention)

Indicator(s)

  • Identified rationale for the classes
  • Identified real or potential strengths and shortcomings of classes
  • Existence of risk-management strategies to mitigate effect of disadvantages
  • Views of program staff

Methods

  • Document review
  • Interviews

Overall, managers and finance personnel tend to identify more advantages than disadvantages of the class approach to grants and contributions, and are much more likely to be of the view that the advantages outweigh the disadvantages.

Both managers and finance personnel frequently point to the efficiency and administrative ease of the classes as their primary advantage, as the classes reduce the need to develop new TB submissions and terms and conditions for every new grants and contributions program. This too is a key advantage highlighted in Environment Canada's Manager's Guide to Grants and Contributions, which notes that the classes were "designed as a mechanism to reduce administrative burden and has served both Environment Canada (EC) and the TBS well."[44] Managers also frequently point to the classes' flexibility to respond to departmental priorities, their ability to leverage resources from other organizations and the nationally consistent approach to their management, including the oversight function by Finance.

One of the key disadvantages of the classes was seen by managers to be the difficulty in aggregating agreement outcomes at the level of the classes or by departmental priority. Similarly, some managers note the need to improve results measurement to demonstrate the achievement of broader outcomes and the need for more strategic planning of contributions to ensure contributions are geared toward desired results.

Interestingly, the breadth of scope of the classes was identified as both an advantage and disadvantage. Both managers and finance personnel note that this breadth of scope means there are numerous inexperienced managers trying to access the classes who may not fully understand the mechanics of establishing and managing class agreements and so require more management oversight (i.e., for the approval and review of class agreements and development of tools to support their use) by a narrow group of managers and finance personnel. Nonetheless, this breadth of coverage was more likely to be seen as an advantage than a disadvantage of the classes as they provide broad access to an important tool for program managers. As one finance respondent remarked, the breadth means that:

"every program has the option of including an outside party and this far exceeds any drawbacks, because if we are able to enable a small group to initiate an undertaking, they will grow and expand and we will all be better off ... that is what G&C is about, enabling them to do something that will further our policy."

Recipients, for their part, were also more likely to identify strengths of the classes than weaknesses. Among the most frequently mentioned strengths, recipients noted that the classes:

  • foster partnerships, and in so doing, provide access to broader networks in order to achieve common goals, a means for the Department to leverage its resources to yield greater results, and access to expertise to deliver programming better than it could otherwise;
  • support high-quality and important projectswhich further EC's mandate and focus on important public policy issues; and
  • are well-designed and delivered by providing the flexibility to adapt elements of funding to project requirements, clear processes and an appropriate reporting burden, efficient disbursement of funds once the funding agreement is signed, and expert support from departmental managers.

The most frequently identified weaknesses of the classes were felt by recipients to be:

  • the timing of and delays in funding;
  • the short timelines for agreements (one year) for projects which exceed (i.e., ongoing work) or are poorly suited (i.e., seasonal) to a fiscal-year funding timeline; and
  • other aspects of the terms and conditions of the classes which require adjustments to project proposals, such as: intellectual property provisions that give the Department rights to the data and to modify reports where project work also supports graduate theses;[45] the precision around timelines and deliverables which make it harder to clearly distinguish agreements from contractual arrangements; and detailed reporting requirements.

A variety of other weaknesses and suggestions for improvement were also offered, although none were raised by more than one recipient respondent. They include: the preponderance to use contribution agreements instead of grants such that funding is lost to support overhead costs; the high proportion of overall project costs that recipients are required to leverage;[46] difficulties in linking project findings to broader federal tracking and reporting mechanisms; and inadequate funding in light of project requirements (i.e., for contribution agreements in support of Canada's obligations).

Summary: Overall, managers and finance personnel tend to identify more advantages than disadvantages of the class approach to grants and contributions, and are much more likely to be of the view that the advantages outweigh the disadvantages. Both the advantages and disadvantages of the classes were typically felt to involve their:

  • administration - efficiency in developing new programming, a consistent national approach, oversight by Finance, and good delivery versus more resources required to oversee a broadly accessible program; and
  • design - flexibility to respond to priorities, leveraging resources, breadth of scope, accessibility, partnerships versus difficulty in aggregating agreement outcomes, and timelines that are poorly suited to ongoing and seasonal projects.

4.3.3.1 Impacts on Evaluation, Accountability and Reporting

When asked specifically about what impact a class approach to grants and contributions has on evaluation, accountability and reporting, several managers were of the view that these processes operated in much the same way as individual grants and contributions programs. Some managers suggested that these activities may even be better in the context of the class. One manager notes that an internal expertise has evolved from repeatedly working with the classes, while another suggested that evaluation, accountability and reporting should be easier because the classes employ a consistent national approach.

More often than not, however, respondents suggested that these processes are more difficult for classes because of the range of organizations and activities supported through the classes. In terms of evaluating the classes, one finance respondent suggests that the effectiveness of the classes cannot be demonstrated because of the scope of the classes and the many program managers involved with them, while a manager notes that these evaluations must rely on qualitative information. One manager in particular questions the need to evaluate the classes from a result-based perspective at all, and wonders whether it makes sense to do so, considering that the classes were designed only to address authorities, delegation, and maximum spending levels. As such, the outputs or outcomes are at the individual agreement level or with the programs to which the individual contribution agreements contribute.

Similarly, one finance respondent notes that accountability at the level of the class is equally difficult given the many individuals involved with the class G&Cs and the absence of any one individual with responsibility for reporting on these class authorities. Other respondents point out, however, that there are appropriate accountabilities at the level of the individual agreements and that financial accountabilities for the review of agreements ensures risks are managed.

Respondents' views of the impact of the class approach on reporting are more varied. The breadth of scope of the classes is also perceived to have an effect on reporting, considering the difficulty in aggregating results given the variability of the projects. The wide scope of the classes may also underlie a one-size-fits-all approach to reporting, which results in an increased workload for managers, and the absence of standard indicators by which managers could measure project success. Other respondents, however, felt that the standard tools and processes for class G&Cs should make it easier to report results in a consistent manner and adapt indicators to different projects.

Although not necessarily a consequence of the class design per se, the broad nature of the classes and unclear accountability at the class level may underlie the absence of a centralized information management system to track class agreements. As noted in Section 2.2, the absence of a central information system means that agreement files are maintained in different areas of the Department, with contribution agreements and financial information stored by regional Managers of Financial Services (MFSs) and project information (e.g., proposals, progress reports) kept by program managers. This poses a significant barrier to the proper evaluation of the classes, as agreement files are very difficult to access. Furthermore, while the Finance Branch maintains records of G&C spending throughout the Department, these records focus primarily on financial information to the exclusion of other information which would improve the ability to evaluate and measure performance, such as project-level indicators or recipient contact information.

Summary: A minority of respondents were of the view that relative to unique G&C programs, evaluation, accountability and reporting are easier under the classes as a result of the nationally consistent approach and expertise that has evolved for their management. More frequently, however, the broad nature of the classes was felt to diminish the ability to evaluate and report on them (i.e., outcomes are too disparate to be sensibly aggregated) or to assign clear accountability for the class authorities. The absence of a centralized information management system for the classes further exacerbates the difficulties in evaluating the classes and measuring their performance.

4.4 Design and Delivery

Overall Findings

The class G&Cs have been developed in compliance with the Transfer Payment Policy and appropriate tools and processes are in place to ensure their delivery is consistent with their design. There may be room to improve oversight of funded projects in terms of proposal reviews and output/outcome reporting.

There is general agreement that the classes are being delivered as designed, as they provide flexibility to respond to priorities (felt to be a key aspect of the classes' design) and are well delivered to recipients. Some concern was voiced about the lack of consistency with which different processes (e.g., governance, risk management, performance measurement, reporting) are implemented across regions within the Department.

Although managers and finance personnel generally expressed a strong familiarity with most existing processes for the management and monitoring of the classes, many managers feel there is a need to make adjustments to processes related to financial controls, performance measurement, and communications. In particular, the project approvals process is perceived to be burdensome and inefficient, and to be causing delays that may impede project success, hurt stakeholder relations and reduce the flexibility of the classes to respond to priorities.

Environment Canada has developed tools to ensure class contribution agreements identify indicators, deliverables and expected results. Evidence from key informants and the file review suggest that these are nearly always identified, although in some cases they may not have been required or appropriate. The nature of the identified indicators varied and was more often linked to output-based information rather than to outcomes, although both types of indicator were frequently identified.

Tools and guidelines have also been established within the Department to ensure that performance information is collected and reported for class agreements. Evidence from key informants and the file review suggest that this information is being collected for most agreements although it can vary from output- to outcome-based information, depending on the nature of the agreement. While performance reporting was not a requirement of the grant or contribution agreement in some cases, such information was not available for one in five agreements reviewed and for which it was reasonable to expect performance reporting.

Although performance information is consistently reported at the agreement level, the broad and heterogeneous nature of the class agreements meant that no systematic process for aggregating or reporting this information to senior decision-makers was in evidence. Given the original intent of the class management approach (i.e., to simplify management of G&Cs and provide flexibility to respond to opportunities), the absence of a systematic performance reporting process at the class level is not surprising.

Accountabilities related to the approvals, financial oversight and management of class agreements are clearly articulated and documented, and are generally understood by most internal key informants. The broad nature of the classes meant that accountabilities for overall class results are much more difficult to identify.

4.4.1 Design and Delivery of the Class G&Cs

Evaluation Issue 8: Is the class G&C delivered as designed? (Progress made, attention needed)

  • Are processes in place to manage and monitor the application of the ARAF at the level of the individual class?

Indicator(s)

  • Demonstration of whether the program/project was implemented as designed
  • Views of program staff and recipients
  • Degree to which T&Cs outline processes to manage/monitor implementation of ARAF

Methods

  • Document review (including T&Cs for classes and ARAF)
  • File review
  • Interviews

The Treasury Board Policy on Transfer Payments which was in effect during the timeframe for this evaluation[47] requires departments to "develop policies and procedures for adequate monitoring of results achieved under contribution agreements and for obtaining suitable information from recipients and from third parties delivering programs to ensure departmental accountability."[48] A review of program documentation demonstrates that the Department has developed tools and guidelines to be followed in the management of G&C agreements, including those signed using class G&Cs authorities, which are encapsulated in Environment Canada's Manager's Guide to Grants and Contributions (June 2004). The Manager's Guide provides overviews and instruction on how managers are to comply with the various departmental G&C processes and procedures, including project eligibility and selection, implementation, monitoring, reporting and termination of a class G&Cs agreement. The guide also outlines the broader mechanisms and processes in support of individual agreements, including: the approvals processes; delegations of signing authorities; the terms and conditions of each of the four class G&Cs; an Accountability, Risk and Audit Framework (ARAF) for the classes; and roles and responsibilities of senior management, delegated program managers, finance representatives, and internal audit and evaluation functions.

In 2005, the Follow-up to the Audit/Evaluation of Class Grants and Contributions report found that "solid progress has been achieved with respect to the management of the Department's grants and contributions.". However, the auditors concluded that there was room for further improvements to the management framework. In particular, program managers had indicated a need for more precision and direction with respect to the management of G&Cs.[49]

The majority of managers and finance personnel were in agreement that the classes are being delivered as designed. While finance respondents most often pointed toward the development and use of various tools and processes (notably the Manager's Guide, but also templates, databases and regular meetings of financial managers) as evidence of the consistency of delivery and design, managers are most likely to refer to their general experiences with the classes' delivery (e.g., "Just in my experience of how we tend to use them ...").

A few respondents from both groups also make specific reference to the classes' flexibility (i.e., broad coverage of departmental priorities, reduced need to develop new terms and conditions, and the ability to quickly shift money to address new or changing priorities), which they feel is a key component of the class design.

Likewise, feedback from funding recipients suggests that the classes are being well delivered. Large majorities of recipients reported that they had been informed at the outset how the funding process would work (primarily through telephone, in-person or e-mail contact with departmental personnel), that the process of obtaining funding unfolded as they were informed it would, and that the terms and conditions of the agreements were well suited to their projects with no need for adjustment. Many of these respondents volunteered that the process and the funding agreements themselves were clear and straightforward, that Environment Canada staff were easy to work with and helpful in terms of explaining the process, and that the program was well administered and communicated. In one respondent's experience, Environment Canada is one of the better departments for communications and support, while another expressed the view that the funding process strikes the right balance between accountability and flexibility.

Evidence from a review of class-agreement files supports the notion that individual agreements are being managed in accordance with the classes' design, although there may be room for improvement in this regard. Contribution agreements were available for all projects for which a formal agreement was required and nine in ten files (88%) contained records related to the financial aspects of the files (see Section 3.2.2 for a detailed discussion of the contents of agreement files). Information related to the review of project proposals and project reporting was less widely available. Specifically, 70% of files included information related to the proposal review process but only 32% contained a copy of the proposal itself. As well, for approximately 20% of files where there was a reasonable expectation of activity, output or outcome reporting, none was in evidence. While there is some question as to whether all administrative information for each class contribution was available for review (see Section 3.2.4 for a more detailed discussion), there would nevertheless appear to be some room to improve the monitoring of class agreements with regard to proposal reviews and output/outcome reporting.

Although generally of the view that the classes are delivered as designed, several managers nonetheless noted an inconsistency of delivery across regions and across internal processes (e.g., governance, risk management, performance measurement, reporting), which are felt to be either too rigid or overly lax. One other delivery issue concerns the difficulty in establishing multi-year agreements.

Of the few recipients who mentioned issues concerning the delivery of the class agreement, one reported confusion about how project funds should be distributed between project partners; another felt the negotiations for their agreement were difficult given the many complexities and restrictions; and two others reported that the process was never well formalized and no formal information was received. These views are clearly in the minority, however.

Summary: The class G&Cs have been developed in compliance with the Transfer Payment Policy (2000) and appropriate tools and processes are in place to ensure their delivery is consistent with their design. There may be room to improve oversight of funded projects in terms of proposal reviews and output/outcome reporting.

There is general agreement that the classes are being delivered as designed, as they provide flexibility to respond to priorities (felt to be a key aspect of the classes' design) and are well delivered to recipients. Some concern was voiced about the lack of consistency with which different processes (e.g., governance, risk management, performance measurement, reporting) are implemented across regions within the Department.

4.4.1.1 Awareness and Perceptions of Management Processes

Collectively, managers and finance personnel expressed a strong familiarity with most existing processes for the management and monitoring of the classes, variously mentioning processes related to approvals (discussed below), financial controls, monitoring and oversight, performance measurement, and communications. To a lesser extent, managers also commented on processes related to the solicitation of proposals and governance and accountability.

When commenting on the consistency in the application of various processes related to the administration of the classes, several respondents from both groups made reference to the Manager's Checklist as a key tool to provide a process overview and general direction for the management of contribution agreements, including a checklist to ensure all necessary work is completed. Results of the file review further suggest that such tools are widely available to managers throughout the Department, as a large proportion of the agreement files examined (70%) included information on the proposal review process. For example, the Grant and Contribution Approval Form was widely used and some files also included the G&C Management Checklist. Furthermore, finance respondents noted that a G&C committee was created to establish priorities and recommendations regarding funding allocations and that a department-wide G&C angiogram exercise[50] was completed to explore ways to streamline the review, approval and overall management processes related to G&C programs.

Although generally felt to be clear and effective, several managers pointed out weaknesses in the processes related to financial controls, performance measurement, and communications which tend to contrast with the generally positive assessments of finance respondents:

  • Financial controls. Following project approvals, a financial review of all agreements is undertaken to enhance the consistency of agreements and ensure adherence to the Treasury Board Transfer Payment Policy. In the view of one finance respondent, this review is particularly necessary for class agreements due to their broad nature and accessibility to managers who make infrequent use of these tools. Another finance respondent notes that the financial review process encompasses the risk management of the classes from a financial perspective, although the management of program risks (e.g., honouring the terms of an agreement) is the responsibility of program managers.
    While several managers felt current financial controls were quite strong and appropriate, one manager pointed out that various financial measures that have been put in place have slowed down the overall the process of delivering funding through the classes.

  • Communications. Some managers note that their own communications with project proponents and stakeholders is good (an opinion echoed by many recipients), but others point to deficiencies in communicating internally about management processes or the outcomes targeted and/or achieved through the classes.

  • Performance measurement. One finance respondent noted that the department has made strides to ensure performance measurement and reporting is ingrained in the management of grants and contributions programming, including audits and evaluations,[51] the requirement for agreements to identify performance indicators (as outlined in the Manager's Guide), and the Transfer Payment Policy which stipulates the need for performance measures and indicators in agreements. This respondent notes that the onus is on the individual manager to respond to these requirements, as planned performance measurement activities are not reviewed as part of the financial review of agreements.

    Several managers agree that performance measurement of individual agreements is carried out by the responsible managers, yet indicate that this is challenging given the resources required for more in-depth performance measurement and the timelines required to assess progress toward outcomes. One manager also notes that the approach to evaluating the classes themselves does not make sense because it fails to take into consideration all activities and resources (e.g., salary, O&M, capital and Gs&Cs) within a given results area. In this way, the evaluation of the classes on their own is "...like you're only looking at the sleeve and you wanted the whole coat."

  • Governance and accountability: One finance respondent points out that the class grants and contributions lack clear governance and accountability, as there is no one senior manager accountable for the achievement of program objectives. In this respondent's view, however, this reduced accountability is a reasonable trade-off for the flexibility afforded by the classes, which is the class tool's greatest asset.

Summary: Although managers and finance personnel generally expressed a strong familiarity with most existing processes for the management and monitoring of the classes, many managers feel there is a need to make adjustments to processes related to financial controls, performance measurement and communications.

4.4.1.2 The Approvals Process

Regardless of their views of the classes' delivery, numerous managers and finance respondents indicated that the departmental G&C approval process negatively affects it. In particular, most managers note that funding decisions are being made too late in the year and that this in turn jeopardizes the ability of project proponents to deliver all project objectives (particularly for seasonal projects) or leads to project cancellations. Other negative consequences of delays in approvals and funding include a reduced likelihood of proponents applying for departmental funding in the future, the need to adjust project designs, and strained relations with funded stakeholders. Other than funding delays, problems with the approvals process were also felt by some to include a greater centralization of the decision-making process that contributes to inefficiencies, and that the same approvals criteria are used for all projects regardless of size, and so are too onerous for smaller agreements.

While acknowledging the utility of the approvals process to provide close scrutiny of all funding proposals and to better align and leverage the contribution investment to priority areas in the wake of expenditure review, finance representatives also note that the approvals process is seen by departmental managers as overly bureaucratic and admitted that it is likely causing serious delays in approving, signing and funding agreements. In particular, these respondents felt delays have reduced the flexibility of classes to respond to opportunities as they arise and to engage in multi-year partnerships, and that the delays have also hurt relationships with stakeholders. One manager questioned the need for such detailed scrutiny of G&C spending, particularly in the case of multi-year funding agreements where this review would already have taken place. While acknowledging the need to administer agreements with proper accountabilities, reporting and evaluation, this respondent is of the view that this level of review is not warranted and that approvals should once again be delegated to the management boards.

Recipients also reported numerous project impacts as a result of delays in receiving funding, with several remarking that these delays occurred after the funding had been approved and so suggesting that the financial review of the agreements was also a likely cause of delay. The most frequently mentioned impact was that it forced recipients to cash-manage projects until funding arrived, which is something that several recipients note was the only means of ensuring their time-sensitive project objectives were achieved (e.g., ongoing or seasonal work). This cash-management approach, however, sometimes meant that funding was diverted from other project activities, which could not proceed until the funds were received.

For those recipients who were unable to cash-manage (e.g., no other project partners), the projects themselves were jeopardized or readjusted (revised the nature of the work, budget or payment schedules) and in two cases, the recipients or their affiliates were required to institute lay-offs.

Summary: The project approvals process is perceived to be burdensome and inefficient and to be causing delays that may impede project success, hurt stakeholder relations and reduce the flexibility of the classes to respond to priorities.

4.4.2 Performance Measurement and Reporting

4.4.2.1 Identification of Indicators, Deliverables and Expected Results

Evaluation Issue 9: Does the class G&C identify clear deliverables and expected results? (Achieved)

  • Do funded agreements clearly identify deliverables and expected results?
  • To what extent are funded agreements targeted toward the achievement of common objectives?

Indicator(s)

  • Demonstration of the class's/project's expected deliverables and results
  • Consistency of objectives among funded agreements
  • Views of program staff and recipients, and unsuccessful applicants

Methods

  • Document review
  • File review
  • Interviews

The Treasury Board Policy on Transfer Payments, which was in effect during the timelines for the current evaluation, specifies the need for all contribution agreements to identify the expected results to be achieved through the agreement.[52] Furthermore, previous audit and evaluation reports of the class G&Cs identified a need to make more visible the linkage between grants and contributions and the realization of program and business line goals and objectives (now board results and project outcomes). The previous auditors had recommended that the Financial Planning and Resource Analysis Division, in its capacity to provide guidance for the Department's planning and reporting, assist boards to implement a process to clearly align G&C results with board objectives.[53]

To this end, Environment Canada's Manager's Guide to Grants and Contributions outlines various guidelines and templates to ensure that "all terms and conditions are addressed when using the G&C authority under the class management or omnibus." In particular, both the terms and conditions for each class, as well as the contribution agreement templates, require managers to identify the activities, purpose and expected results of the agreements and to demonstrate how these are linked to departmental priorities.[54]Furthermore, an Accountability, Risk and Audit Framework (ARAF) was developed specifically for the class G&Cs and includes a logic model and associated definitions to outline what is meant by activities, outputs and outcomes.[55]

Virtually all managers, finance personnel and recipients felt that funded agreements clearly identify deliverables and expected results. Numerous respondents from all groups, however, felt that the results and associated indicators tended to identify activities and outputs or in some cases a mixture of outputs and outcomes. Further, in a minority of cases, managers and recipients pointed out that indicators and outcomes may not have been developed at all, as this was neither possible nor appropriate (e.g., research projects) or not a requirement at the time the agreement was signed (although it is now).

The results of the file review are consistent with feedback received from key informant interviews and reveal that all agreements identified project objectives. Roughly half (49%) of the class agreements reviewed identified objectives that were both output- and outcome-related, while over one-third (37%) identified primarily output-related objectives (e.g., objectives focused on activities and immediate outputs such as "collect data on meteorological events" or ‘training"), and 14% have outcome-related objectives only (e.g., objectives that include the result of the activities or outputs, such as "improved weather prediction for safety" or training "to improve community knowledge and participation").

A key challenge in the identification and measurement of performance information for class G&Cs concerns the inability to link agreement indicators and results to a small set of shared outcomes to enable the aggregation and reporting of performance information at the class level. As discussed in detail in Section 3.2.4, this is not possible to do for classes, given the vast array of shorter and medium-term departmental outcomes to which the class agreements are aligned. This is further exacerbated by two issues: a large proportion of agreements support projects designed to result in outputs or activities rather than outcomes; and the requirement to link project results to departmental priorities, without a corresponding requirement to identify the lower-order departmental outcomes on which the project is more likely to have an immediate and measurable effect.[56]

In light of these issues inherent in the design of the classes, and the commensurate importance of being able to demonstrate the value of the classes while ensuring flexibility and broad access to grants and contributions as a management tool, a more strategic approach to the evaluation and performance measurement of the classes is required.

Summary: Environment Canada has developed tools to ensure class contribution agreements identify indicators, deliverables and expected results. Evidence from key informants and the file review suggest that these are nearly always identified, although in some cases they may not have been required or appropriate. The nature of the indicators identified varied and was more often linked to output-based information rather than to outcomes, although both types of indicator were frequently identified. The broad nature of the classes warrants a more strategic approach to performance measurement and evaluation.

4.4.2.2 Collection of Performance Information

Evaluation Issue 10: Is performance data for class agreements collected against activities/outcomes? If so, is collected information used to inform senior management/decision makers? (Progress made, Attention needed)

  • To what extent is performance information aggregated to demonstrate progress toward specific intermediate outcomes?

Indicator(s)

  • Presence of populated performance data system
  • Decisions based on performance information
  • Views of Finance and program staff and recipients

Methods

  • Document review
  • Interviews

Treasury Board policy requires that managers regularly monitor and manage the progress and activities of recipients for the purposes of performance measurement as well as financial purposes.[57] To this end, Environment Canada's internal processes for the management of G&C agreements requires that each contribution agreement include a clause stipulating that funding recipients will provide progress reports pertaining to their project activities[58]and that performance measurement and criteria to mitigate risk are to be put in place at the individual class G&C agreement level.[59]

In line with the requirements of the Transfer Payment Policy, nearly all managers and recipients report that performance information is collected and reported for their agreements, and in all cases final deliverables were prepared and submitted. Those few recipients who indicated that they had not collected performance information noted that their project was ongoing or that reports related to their funded projects (e.g., workshops) were developed by their federal partners.

In terms of the types of performance information collected, managers reported that the nature and quality of the performance data varied. Similarly, recipients who described the types of performance information collected were equally likely to point to outcome and output measures, the latter being particularly true of agreements to support such work as basic research or grants. This too is consistent with departmental expectations for performance measurement. Documentation shows that managers are required to demonstrate that the outputs of agreements measure the value of the individual contributions in supporting specific departmental results and priorities, even though higher level indicators may not be in place.[60]

Evidence from the file review also suggests significant variation in the nature of activity and outcome reporting. As per internal processes for the management of class agreements (mentioned above), contribution agreements were typically found to include a clause related to reporting, with payments being linked to deliverables and completion of project activities. This clause requires that recipients provide financial information and reporting covering the activities to be undertaken by the recipients under the contribution agreement, but because activities funded by class grants and contributions varied significantly, so too did the format and content of activity and outcome reporting. Examples included:

  • Written interim, progress and/or final reportsproduced specifically for the purposes of the agreement;
  • Reporting templates, which provided guidance for recipients to report on activities and progress, issues and challenges and results, among other areas;
  • Conference proceedings/published papers; and
  • Correspondence/annual reports/meeting summaries, generally found in cases where the class agreement funded the operation of an organization or entity.

Overall, 64% of the files included some form of activity or outcome reporting. Specifically, these files contained either: final reports (42%); progress, interim or activity reports (38%); an annual report (17%); correspondence (18%), meeting/committee materials (14%); or other materials (22%). A small minority of files (3%) contained documentation relating to an evaluation of the project.

For the minority of class agreement files that did notcontain activity, output or outcome reporting, in roughly 15% of cases the agreement had no requirement for reporting (e.g., reporting is not required for Graduate Supplements or assessed contributions, such as memberships in international organizations). For the remaining 20% of files overall, where there was a reasonable expectation of activity, output or outcome reporting, none was evident in the files.

Summary: Tools and guidelines have been established within the Department to ensure that performance information is collected and reported for class agreements. Evidence from key informants and the file review suggest that this information is being collected for most agreements, although it can vary from output- to outcome-based information depending on the nature of the agreement. While performance reporting was not a requirement of the grant or contribution agreement, in some cases, such information was not available for one in five agreements reviewed and for which it was reasonable to expect performance reporting.

4.4.2.3 Reporting of Performance Information

As noted earlier in the discussion of project outcome achievement (Section 4.2.1), a 2005 follow-up to earlier audits and evaluations of the classes[61] concluded that although an ARAF was completed for class G&Cs, the reporting obligations on specific class contributions remained imprecise. Specifically, not all G&Cs programs were capturing results-based data (in a systematic manner) and reporting them to departmental senior management.

In terms of reporting performance information stemming from individual agreements, most managers and nearly all recipients mentioned that a written report for their project was submitted to Environment Canada. Those few recipients who made no specific mention of a written report were all involved in agreements that were jointly administered by Environment Canada and indicated that the Department would have prepared a report. Other means used by several funding recipients to relay project information to Environment Canada include meetings and Web postings of reports.

Managers were divided in terms of whether performance information related to funded projects was used to inform senior decision-makers in the department. Of those who reported using it to inform senior management, most remarked that this was typically done in the context of reporting on the programs supported by the class agreements. Others noted that when used to inform senior decision-makers, this was typically for the purpose of evaluating whether the Department was receiving value from the funded work and, in one instance, to make the case for a multi-year funding agreement. Similarly, finance personnel note that most such reporting is very high-level, such as reporting through the Departmental Performance Report (DPR) and publishing agreements over $100,000 on public accounts (as required by the Transfer Payment Policy),[62] and through the Results Management Tool, which is no longer used but had performance data on outcomes linked to the G&C portion of the budget.

When asked whether the performance information was aggregated to demonstrate progress toward specific departmental outcomes, managers were more likely to indicate that it was not. Most of these managers suggested that the performance information from agreements was not amenable to this type of aggregation, since the project outcomes are so diverse. One manager, however, suggested that this aggregation should be possible if it were to focus on performance information from agreements within a specific program area. Of those managers who reported that this aggregation of performance information does take place, one noted that the information was primarily subjective while several others pointed out that the aggregation of information was focused on a given program area (and in some cases, rolled up with the results of the broader departmental program) and that the focus was more on activities than project outcomes. Considering that the original intent of the class management approach was not to facilitate performance reporting but rather to simplify and facilitate the management of G&Cs and give the Department ongoing flexibility to respond in a timely manner to new opportunities, it may not be surprising that no systematic means of aggregating and reporting on performance at the class level is in evidence.

Summary: Although performance information is consistently reported at the agreement level, the broad and heterogeneous nature of the class agreements meant that no systematic process for aggregating or reporting this information to senior decision-makers was in evidence. Given the original intent of the class management approach (i.e., to simplify management of G&Cs and provide flexibility to respond to opportunities), the absence of a systematic performance reporting process at the class level is not surprising.

4.4.3 Accountability

Evaluation Issue 11: Who is accountable for the class G&Cs? Are the roles and responsibilities of all groups involved clear? (Progress made, attention needed)

  • Is there an appropriate accountability framework?
  • Are there any factors which impede or enhance accountability for the classes?

Indicator(s)

  • Defined and known program management structure for the classes
  • Views of Finance and program staff, recipients, and unsuccessful applicants

Methods

  • Document review
  • Interviews

The Treasury Board Policy on Transfer Payments outlines departmental responsibilities and accountabilities related to transfer payment programs and requires that departments establish policies and procedures to ensure that effective financial and program controls are designed and implemented.[63] These controls are meant to ensure:

  • the exercise of due diligence in the selection and approval of recipients and the management and administration of the programs;
  • the existence of efficient and effective accounting and other procedures;
  • proper program and accounting records and other relevant documents are maintained;
  • a results-based management and accountability framework is prepared to measure and report on results; and
  • that departmental capacity exists to effectively deliver and administer the transfer payment programs.

Environment Canada's Manager's Guide to Grants and Contributions provides information regarding the roles and responsibilities of various groups tied to the class G&Cs, including program managers, finance, recipients, as well as the audit and evaluation functions. Overall, the guide outlines delegated signing and approval authorities for different agreement values (in accordance with Section 5.1 of Appendix A of the departmental Financial Signing Authorities) and defines the roles and responsibilities of managers, funding recipients and various departmental groups in the authorization and payment of eligible expenditures, monitoring and reporting of class agreements.

The Accountability, Risk and Audit Framework (ARAF) for the Departmental Class Grants and Contributions provides more information related to roles, responsibilities and relationships of senior management, delegated managers, and audit and evaluation.[64]Senior management sets out directions, priorities, strategies, resource requirements and performance measures for the delivery of respective key results and ensures horizontal management of issues. Delegated managers are responsible for front-line delivery of departmental programs and initiatives through their work-related activities. The class Terms and Conditions permit managers to enter into transfer payment agreements as one mechanism to achieve results and carry out program delivery.

Currently, within the new departmental structure, OPGs are to present their preliminary G&C plans (allocations) to their respective boards. Boards are then to exercise due diligence to review allocations and implications for results (based on priorities). A G&C Review Committee (struck in the winter of 2007) is available to assist with analysis. The objective of the G&C Review Committee is to ensure that Environment Canada effectively uses grants and contributions to achieve departmental results; it was not established to review the G&C proposals themselves. The mandate of the committee is to:

  • Act as the policy lead for G&Cs within the Department;
  • Provide analysis and advice to support Board decision-making;
  • Make recommendations to EMC regarding existing and future funding levels;
  • Manage a $1-million central reserve for strategic opportunities;
  • Conduct a mid-year review of G&Cs within the Department;
  • Provide recommendations on signing authority levels for G&Cs; and
  • Oversee ongoing management under the G&C framework.[65]

Most managers and all finance personnel feel they have a clear understanding of the roles, responsibilities and accountabilities of the class G&Cs, with finance representatives noting that the delegated signing authorities are documented and that procedures and steps for approvals are known. Managers, for their part, point to their familiarity with the accountabilities surrounding approvals, monitoring and oversight, and performance measurement and reporting of class agreements. Similarly, nearly all recipients reported that they clearly understood their roles and responsibilities under the G&Cs agreements and had a clear sense of the roles and responsibilities of Environment Canada and other parties involved in their agreements.

Several managers and finance personnel remarked that the department's focus on accountability through labour-intensive approvals and financial review processes have led to delays in funding agreements (and so jeopardized the Department's accountability to recipients) and a lack of clarity concerning the purpose of multiple financial "sign-offs." One finance respondent pointed out, however, that Board oversight and approvals ensures that funds are used in the best possible manner.

A number of factors that affect accountabilities for the class grants and contributions were identified in the interviews with managers and finance personnel:

  • Duplication. Several managers suggest there is a lack of clarity between the roles of managers and finance personnel, and that this may lead to a duplication of effort in the roles assigned these groups for monitoring agreements and approving payments/deliverables. That is, some managers noted that finance personnel would question whether the deliverables received were adequate or appropriate under the terms of the agreement. Managers were of the view that program managers should focus on program-related questions (i.e., deliverables; results achieved to date), while finance personnel should focus only on financial questions (i.e., examining cash flow statements and expenditures).
  • Scope and flexibility of the class design.Several finance personnel note that the broad access and flexibility inherent in the design of the classes make it difficult to clearly define outcomes, to identify a single individual responsible for reporting on the classes' successes, to maintain managers' understanding related to processes and procedures, and to aggregate the results of agreements to demonstrate progress toward departmental results. One finance respondent suggested that accountabilities would be clearer (especially at more senior levels) if the classes were re-designed to cover only OPGs that fall under the responsibility of a single ADM.
  • Changing Context. Some managers and finance personnel indicated that changes to Treasury Board expectations vis a vis reporting on the results of classes, as well as organizational changes within the departments (i.e., need to re-align class G&Cs to current priorities), create accountability challenges as guidelines and processes in support of the class need to be updated. One recipient also noted a lack of clarity around changing departmental priorities and the effects this has on existing projects.

Summary: Accountabilities related to the approvals, financial oversight and management of class agreements are clearly articulated and documented, and are generally understood by most internal key informants. The broad nature of the classes meant that accountabilities for overall class results are much more difficult to identify.

4.5 Lessons Learned

Evaluation Issue 12: What are the best practices and lessons learned from the class G&Cs? (N/A)

Indicator(s)

  • Identified lessons learned and best practices
  • Identified strengths and weaknesses
  • Factors that contribute to/detract from the achievement of results
  • Views of Finance and program staff, and recipients

Methods

  • Document review
  • Interviews

Managers and finance personnel identified a number of best practices and lessons learned relative to the management of the class G&Cs. Key lessons identified through the key informant interviews include the following:

  1. It is important to provide managers with standard guidance materials. Standard information, tools (guides, templates) and processes ensure more consistent contribution agreements across areas of the Department, reduce the need for centralized risk management, and improve managers' effectiveness in dealing with contribution agreements. Furthermore, ensuring that managers understand the value of tools and guidelines may preclude mistakes in the establishment of contribution agreements, facilitate the success of funded projects, and encourage productive relationships with partners.
  2. A dedicated manager is important to providing consistent oversight of contribution agreements. Continual involvement, communication with and monitoring of project proponents ensures that outcomes are being achieved, that there is greater consistency in contribution agreements, and that departmental personnel can exert more influence on research priorities.
  3. A balanced approach to departmental accountability would strengthen program delivery. Appropriate empowerment and delegation of project approvals and oversight would free resources to review newer or higher-risk projects, improve the timeliness of funding allocations, and improve accountability when involvement in approvals and oversight is based on expertise. At the level of the individual agreement, a more strategic focus on accountability and reporting would avoid alienating stakeholders, improve the Department's credibility, reduce project delays, and improve the effectiveness of the classes.
  4. More timely approvals and financial review processes would facilitate the management of class G&Cs and enhance their effectiveness. More timely allocation decisions would enable the Department to plan and manage G&Cs responsibly, achieve program results, plan and manage communications opportunities, and better manage relationships with partners and stakeholders.[66] Key informants further note that a clearer and more straightforward process to release project funds once project approvals have been received would reduce project risks (e.g., risk of project not proceeding as planned) and improve project quality (e.g., allow sufficient time to engage appropriate project personnel).
  5. Senior management oversight ensures that funded projects are consistent with departmental priorities.
  6. Partnerships are important to leverage G&Cs resources, maintain relationships with partners, and advance departmental priorities.
  7. It is important to achieve the right balance of flexibility and accountability in designing and managing the classes. The flexibility of the class G&Cs is important for the Department to meet its business objectives but creates challenges to demonstrate accountability at the overall class program level.
  8. The solicitation of project proposals would strengthen program delivery and success. The solicitation of proposals avoids any suggestion of favouritism in the awarding of agreements and encourages involvement of a wider range of stakeholders in activities related to departmental priorities.

5.0 CONCLUSIONS

The present evaluation of Environment Canada's Class Grants and Contributions focused on the fiscal years 2006-07 and 2007-08. The findings of the evaluation lead to the following broad conclusions about the classes' relevance, success, cost-effectiveness, and design and delivery:

  1. The class grants and contributions are an important tool to encourage the participation of external stakeholders in activities that support departmental priorities. These class authorities account for a large proportion of G&C spending in the Department. Furthermore, their flexibility, broad scope, and wide availability to managers allow the Department to respond quickly and efficiently to opportunities and changing priorities, and to address gaps in G&C support across the Department.
  2. The classes are being used appropriately to support the work of outside parties in furthering departmental priorities. No evidence was found to suggest that the classes are being used to undertake departmental program activities or to avoid the development of individual terms and conditions for unique G&C programs.
  3. The classes are generally well managed and comply with the requirements of Treasury Board's Transfer Payment Policy, which was in effect during the timeframe for this evaluation, although there is room to improve management oversight of the classes. Numerous tools and processes exist to ensure class agreements align with departmental priorities, are used appropriately, and demonstrate the achievement of intended results. Evidence suggests, however, that there is a need to update and refine certain tools and mechanisms to more clearly focus on and articulate current departmental priorities and expectations with regard to performance measurement and reporting, and to streamline and improve the timeliness of the initial approval and financial review stages.
  4. The class G&Cs are a cost-effective means to support stakeholders in the pursuit of departmental priorities. Further efficiencies might be realized by implementing more balanced risk-mitigation strategies in the delivery of these G&Cs, including delegation of approval authorities, clearer accountabilities, and the use of multi-year agreements.
  5. Accountabilities for the classes are clear at the level of agreement approvals and managers' responsibilities in administering individual agreements. The broad nature of the classes, however, means that overall accountabilities are unclear or non-existent.
  6. Existing processes for the management and monitoring of the classes and agreements are generally well understood by program managers and finance personnel, as evidenced by their familiarity with these processes. The mechanics of managing class agreements are less well understood by managers who make infrequent use of these class authorities.
  7. Class-level financial oversight is clearly understood by Finance personnel, although there appears to be a lack of clarity concerning the roles, responsibilities and processes related to initial financial sign-off of approved agreements, which is felt to be overly complex, and the monitoring and approval of project payments, which may lead to duplication. It remains unclear, however, whether the roles played by Finance personnel reflect a necessary oversight function to manage risks associated with so ubiquitous a management tool or if there are means of streamlining and clarifying these processes so as to minimize delays in releasing project funds.
  8. Departmental processes related to project approvals and financial reviews are complex and not strategically focused on project risk. This in turn has led to delays in the approval and distribution of G&C funding to project proponents, has limited the classes' flexibility to respond to opportunities and changing priorities, and has somewhat adversely affected the ability of funded projects to achieve desired results.
  9. The design of the classes, and in particular their broad scope and flexibility, presents challenges for performance measurement, evaluation and reporting at the level of the classes. Performance measurement is occurring consistently at the individual agreement level and the classes themselves are generally believed to be contributing to the achievement of departmental priorities. Nonetheless, the variety of outcomes supported through class agreements has meant that no systematic process exists to aggregate and report on project results in order to demonstrate their combined contribution to the achievement of departmental priorities at the class level.
  10. The absence of a centralized information-management system for the classes exacerbates difficulties in measuring their performance and evaluating them. Information related to the performance of individual agreements is maintained in a decentralized fashion and so cannot be easily aggregated to demonstrate performance in terms of the achievement of deliverables, activities and outputs. This also poses significant barriers to accessing key information for the evaluation of these class authorities (e.g., lists of recipients, project files).

6.0 RECOMMENDATIONS

  1. The IS Board should clarify overall accountabilities for the class G&Cs. One option to achieve more clarity in this regard would be to re-align the classes according to the Department's new program activity architecture (PAA) and strategic outcomes. In this way, overall accountability for each of the classes would rest with the senior manager who wields primary responsibility for the departmental activities that support each of the Department's strategic outcomes.
  2. The IS Board should clarify the respective roles, responsibilities and accountabilities of managers and finance personnel in the financial review, management and oversight of funded agreements. Greater clarity may in turn lead to process efficiencies by avoiding duplication in initial financial approvals and the management of project deliverables and payments.
  3. The IS Board should explore means of streamlining the G&C approvals process. To this end, consideration should be given to making the process more risk-based, providing early approvals for less discretionary G&C spending, implementing delegated authorities and incorporating the use of multi-year agreements where appropriate.
  4. The IS Board should develop a more strategic approach for ongoing performance measurement of class G&Cs.This approach should consider linking unique project outcomes, activities and/or outputs with shorter-term departmental outcomes to which an attributional link is more easily made and to more clearly outline each project's logical association with the Department's strategic outcomes.
  5. The IS Board should update standardized tools for managers to ensure they are aligned with the current departmental structure, reflect the needs of the new Policy on Transfer Payments, and are employed in a consistent manner across programs and regions. In updating these tools, the IS Board should consider:
    • the development of a standard training module for managers wishing to make use of classes in order to improve the quality of agreements and so reduce the burden on finance personnel and improve the efficiency of the financial review process;
    • language which encourages the identification of clear and measurable outcomes for performance-measurement purposes in instances where the collection of outcome-related information is feasible within the timeframe and financial scope of funded projects; and
    • the development of standard indicators to guide managers and project proponents in the measurement of project success.
  6. The IS Board should develop a centralized information-management system to track financial and performance information for class G&C agreements, to facilitate the aggregation and reporting of this information to senior departmental decision-makers, and to improve the accessibility and comprehensiveness of this information for evaluation.

7.0 MANAGEMENT RESPONSE

MANAGEMENT RESPONSE

The ADM, Finance and Corporate Branch, and the ADM, Environmental Stewardship Branch, (the ADMs) take responsibility on behalf of all the involved branches for implementing the management response.

The proposed responses to the recommendations draw from the draft EC Action Plan for G & C Reform, which will be presented for approval to EMC on May 27th. It is proposed that the responses outlined below be implemented department-wide (not just by IS Board) as all four boards use classes to fund projects.

Six recommendations were developed for the IS Board based on the evaluation findings and conclusions.

1. The IS Board should clarify overall accountabilities for the class G&Cs. One option to achieve more clarity in this regard would be to re-align the classes according to the Department's new program activity architecture (PAA) and strategic outcomes. In this way, overall accountability for each of the classes would rest with the senior manager who wields primary responsibility for the departmental activities that support each of the Department's strategic outcomes.

The ADMS agree with this recommendation.

Accountabilities for Class G&C programs will be clarified through the development of new Class G&C authorities. It is proposed that these new Classes be aligned to the department's new program activity architecture (PAA) and strategic outcomes. Through this alignment, the operational Boards are responsible for recommending to the Deputy Minister program priorities and the annual budget allocation under the new classes and for monitoring the overall achievement of results. Branch heads and their delegates, supported by Corporate Finance, are responsible for the negotiation, implementation and achievement of results of individual agreements under the new classes.

Responsible Party: Co-lead, ADMs of ESB and Finance; Coordination of the Renewal process will be the responsibility of Board Secretaries (ES/EP/WES)

2. The IS Board should clarify the respective roles, responsibilities and accountabilities of managers and finance personnel in the financial review, management and oversight of funded agreements. Greater clarity may in turn lead to process efficiencies by avoiding duplication in initial financial approvals and the management of project deliverables and payments.

The ADMs agree with this recommendation.

This was also a recommendation of the "Angiogram" work undertaken in fiscal year 2008-09, and will be a key deliverable under Environment Canada's Action Plan for G&C Reform. An important step has already been taken in support of this recommendation with the approval of delegation of authorities by the Minister in March, 2009. Led by Corporate Finance, the ADMs will clarify roles and responsibilities of managers and finance personnel in the development of contribution agreements and the subsequent financial review, sign-off and oversight. A clear, documented and streamlined process will be the result. Training and information for managers and finance officers will ensure the process is widely understood and consistently applied across the department.

In particular, Corporate Finance will:

  • Develop information sessions for G&C program managers and finance officers on: Treasury Board's new Policy on Transfer Payments; EC's G&C Approval Process and EC's Action Plan on G&C Reform
  • Prepare technical training/coaching for MFS's and program managers on new approval processes and updated finance procedures (including checklists and a "G&C 101" internal website)

Expected Completion date: August, 2009 (for clarification of roles & process)
Responsible Party: Lead - Corporate Finance, informed by a departmentally-representative team

3. The IS Board should explore means of streamlining the G&C approvals process. To this end, consideration should be given to making the process more risk-based, providing early approvals for less discretionary G&C spending, implementing delegated authorities and incorporating the use of multi-year agreements where appropriate.

The ADMs agree with this recommendation.

The work referred to in the Management Response to evaluation Recommendation 2 to clarify and streamline the roles and responsibilities of managers and finance personnel in the development, financial review, sign-off and oversight of contribution agreements, will also deal with the initial allocation process and timing of funding decisions and is a key deliverable under the fifth theme of Environment Canada Action Plan for G&C Reform (Achieving Clarity and Consistency in Practices).

Specific deliverables that will respond to this recommendation include the creation and dissemination of step-by-step written guidelines for approval of agreements (which should reduce the number of signatures required), the creation of a standard contribution agreement made available on the internal G&C website, and an update EC Managers' Guide to Gs&Cs. This work to streamline approvals may also be informed by EC's move to a more risk-based approach.

This should result in shorter wait times from application through to approved contribution agreement and reduce frustration by employees and funding recipients due to delays and inefficient management practices

An important step forward was taken for fiscal year 2009-10, when notional G&C budgets for "fenced funds" were allocated to Boards in January, 2009. This has resulted in much earlier allocations of G&C budgets down to the program level. As mentioned above, G&C approvals have also been streamlined with the approval of the delegation of authorities in March, 2009.

The recommendation also refers to the need to build a risk management approach to the approval of Gs&Cs. While a Risk Management framework, along with a suite of risk tools, will be an important theme under EC's Action Plan, the goal of the work is to reduce administrative burden (where appropriate) and not to streamline approval processes.

Expected Completion date: Fall, 2009
Responsible Party: Lead - Corporate Finance, supported by Outreach Division, Environmental Stewardship Branch, and informed by a departmentally-representative team

4. The IS Board should develop a more strategic approach for ongoing performance measurement of class G&Cs. This approach should consider linking unique project outcomes, activities and/or outputs with shorter-term departmental outcomes to which an attributional link is more easily made and to more clearly outline each project's logical association with the Department's strategic outcomes.

The ADMs agree with this recommendation.

The new Class Gs&Cs will be aligned to the department's Program Activity Architecture and Strategic Outcomes, thereby facilitating better performance measurement. Under the new Transfer Payments Policy, a Performance Measurement Strategy (which replaces the RMAF/RBAF requirement of the past) will be developed for each Class. These Strategies will be aligned with the Performance Measurement Framework (PMF) for the PAA Program Activities that are supported by these new class authorities. Each class PMS will outline the logical association between the range of activities potentially supported through the classes and lower-level results and indicators aligned with Program Activity results and to which class agreements can more easily be linked. These common lower-level performance indicators which may be used to measure achievement of results will be used wherever feasible.

In addition to performance measurement at the Class level, Contribution Agreements will specify expected deliverables for each individual project and wherever possible will also identify any measurable project results (i.e., outcomes) that are linked to PAA results (i.e., through common lower-level results identified through the PMS). Project deliverables will provide the basis for payment to the recipient.

As part of the criteria for project selection, the program will ensure that projects are chosen for which there is a strong linkage between the expected project deliverables and/or results and the expected results of the overall PAA program activity. Applicants will be asked to explain in their funding applications what these linkages are and program managers will assess the strength and validity of the proposed linkages based on their knowledge of the subject matter.

Expected completion date: March, 2010
Responsible Party: Co-leads - ADMs Finance and ESB for Renewal of the Classes; Coordination of the Renewal process will be the responsibility of Board Secretaries (ES/EP/WES)

5. The IS Board should update standardized tools for managers to ensure they are aligned with the current departmental structure, reflect the needs of the new Policy on Transfer Payments, and are employed in a consistent manner across programs and regions. In updating these tools, the IS Board should consider:

  • the development of a standard training module for managers wishing to make use of classes in order to improve the quality of agreements and so reduce the burden on finance personnel and improve the efficiency of the financial review process;
  • language which encourages the identification of clear and measurable outcomes for performance measurement purposes in instances where the collection of outcome-related information is feasible within the timeframe and financial scope of funded projects; and
  • the development of standard indicators to guide managers and project proponents in the measurement of project success.

The ADMs agree with this recommendation.

As part of EC's Action Plan for G&C Reform a number of tools are being either updated or created to reflect the requirement of the Transfer Payments Policy (TPP)

Under the Action Plan's Theme 2 (Culture Change) this will include:

  • Information sessions for G&C project officers & administrative support staff on G&C Management. This will reflect, among other updates, EC's Action Plan for G&C Reform, the TPP and the new, streamlined G&C Approval Process for managers.
  • Technical training has aleady been provided to Finance G&C Experts and is being followed up with bi-weekly interactive updates.
  • Training sessions on Risk Management approach (see below)
  • Regular messaging on goals and progress of G&C Reform for Management Council

Under the Action Plan's Theme 4, this will include:

  • A standardized risk management approach and suite of risk assessment tools developed in collaboration with front-line managers

Under the Action Plan's Theme 5, this will include:

  • Step-by-step written guidelines for approval of agreements
  • Standard contribution agreement (template)
  • Updated "EC Managers' Guide to Gs&Cs"
  • Written procedures for direct deposit, cash advances
  • Standard application forms, reports and audit requirements

The ADMs will seek the advice and input of Audit and Evaluation and Corporate Management Directorate to include in these tools, language which encourages managers to include clear and measureable indicators and outcomes in the development of agreements.

Expected completion date: Fall, 2009 for initial round of info sessions & tools; March, 2010 for training; Training of Risk Management tools will begin in 2010-11, following a pilot of the tools in 2009-10
Responsible party: For themes 2 & 5 - Corporate Finance, supported by Outreach Division, Environmental Stewardship Branch; for theme 4 - Co-lead of Outreach Division, Environmental Stewardship Branch and Corporate Finance

6. The IS Board should develop a centralized information-management system to track financial and performance information for class G&C agreements, to facilitate the aggregation and reporting of this information to senior departmental decision-makers, and to improve the accessibility and comprehensiveness of this information for evaluation.

The ADMs agree with this recommendation.

A key deliverable under theme 7 of EC's Action Plan for G&C Reform (Improving Recipient Access and Efficiency Using Technology), is the development of a centralised G&C information management system to track financial, project and performance information. This new system will serve funding applicants and recipients, program managers and finance officers. It will be designed to hold information related to all of Environment Canada's grants and contributions projects and expenditures, including those under the new classes and funding programs.

The first step in developing this system is a business process and needs analysis that will be launched in 2009-10 fiscal year, with an initial focus on the established community funding programs. Funding for this first step has been identified as a priority by the ES Board.

As new Classes are developed over this fiscal year, the information management needs of this type of G&C will be incorporated into the business process analysis.

The result of the work this fiscal year will be a business case for the development of a department-wide system. The investment in such a system will be significant and will be presented for consideration to senior management. In assessing options for the new management information system, we will consider the appropriateness of adopting models used by other government departments.

Expected completion date: March, 2010 (Business case); Target for launch of new system would be 2011-12. If the business case is not accepted, the department would continue to operate existing systems, integrating the new Classes and the new risk tools into these systems.
Responsible party: Outreach Division, Environmental Stewardship Branch lead with CIOB and Corporate Finance.

Annex 1 - Evaluation Issues and Questions

Evaluation QuestionIndicatorData Source
Relevance

1. Is there a need for this class Grant and Contribution (G&C) to support third parties in the pursuit of departmental priorities related to (research and development/sustainable development/international commitments)?

  • To what extent does this class G&C overlap with or duplicate alternative funding sources to support these groups?
  • Why was a class contribution chosen over other funding alternatives?
  • Evidence of similar internal or external G&C programs
  • Demonstration of the utility/rationale for program/ funding mechanism
  • Views of program staff
  • Number of existing contribution programs and agreements that are subsumed within the classes
  • Document review
  • Interviews

2. To what extent does the class G&C support projects that are consistent with departmental priorities?

  • Are the classes used to support agreements aligned directly with program activities?
  • Consistency of agreement with program and departmental objectives
  • Existence of gaps in the coverage of class objectives
  • Reach is analyzed and targeted
  • Both the reach and activities are connected to environmental requirements
  • Views of program staff
  • Proportion of agreements and contribution programs that align directly with program activities
  • Proportion of agreements to be evaluated as part of programs to which they align.
  • Document review
  • File review
  • Interviews
Success

3. To what extent have the intended outcomes of funded projects been achieved as a result of the class G&C?

  • To what degree have funded projects contributed to the achievement of departmental objectives?
  • Evidence of intended output and outcome achievement
  • View of program staff and recipients
  • Document review
  • File review
  • Interviews

4. Have there been any unintended (positive or negative) outcomes at the level of the class G&C? At the level of the individual agreements?

  • Were any actions taken as a result of these?
  • Presence/absence of unintended outcomes
  • Documented management actions and/or lessons learned from unintended outcomes
  • Views of program staff and recipients
  • Document review
  • File review
  • Interviews
Cost-Effectiveness

5. Are there alternative ways of supporting third parties in the pursuit of departmental priorities?

  • If the class G&C continues, how could its efficiency be improved?
  • Analysis of delivery options/opportunities
  • Views of Finance and program staff
  • Document review
  • Interviews
6. Has the class G&C provided value for federal dollars spent?
  • Analysis of costs of program compared with achievement of program outcomes
  • Selection process for areas of importance are applied
  • Document review
  • File review
  • Analysis completed for outcomes question (#3)

7. What are the advantages and disadvantages of using a class approach to support third parties in the pursuit of departmental priorities?

  • Do the advantages outweigh the disadvantages?
  • What are the implications for evaluation, accountability and reporting?
  • Identified rationale for the classes
  • Identified real or potential strengths and short-comings of classes
  • Existence of risk-management strategies to mitigate effect of disadvantages
  • Views of program staff
  • Document review
  • Interviews
Design and Delivery

8. Does the class G&C identify clear deliverables and expected results?

  • Do funded agreements clearly identify deliverables and expected results?
  • To what extent are funded agreements targeted toward the achievement of common objectives?
  • Demonstration of the program's/project's expected deliverables and results
  • Consistency of objectives among funded agreements
  • Views of program staff and recipients, and unsuccessful applicants
  • Document review
  • File review
  • Interviews

9. Is the class G&C delivered as designed?

  • Are processes in place to manage and monitor the application of the ARAF at the level of the individual class?
  • Demonstration of whether the program/project was implemented as designed
  • Views of program staff and recipients
  • Degree to which T&Cs outline processes to manage/monitor implementation of ARAF
  • Document review (including T&Cs for classes and ARAF)
  • File review
  • Interviews

10. Is performance data for class agreements collected against activities/outcomes? If so, is collected information used to inform senior management/decision makers?

  • To what extent is performance information aggregated to demonstrate progress toward specific intermediate outcomes?
  • Presence of populated performance data system
  • Decisions based on performance information
  • Views of Finance and program staff and recipients
  • Document review
  • Interviews

11. Who is accountable for the class G&C? Are the roles and responsibilities of all groups involved clear?

  • Is there an appropriate accountability framework?
  • Are there any factors which impede or enhance accountability for the classes?
  • Defined and known program management structure for program
  • Views of Finance and program staff, recipients, and unsuccessful applicants
  • Document review
  • Interviews
12. What are the best practices and lessons learned from the class G&C?
  • Identified lessons learned and best practices
  • Identified strengths and weaknesses
  • Factors that contribute to/detract from the achievement of results
  • Views of Finance and program staff and recipients
  • Document review
  • Interviews


Annex 2 - List of Background Information and Supporting Documentation

Environment Canada. 2001. Audit and Evaluation of Class Contributions in Support of Environmental & Sustainable Development Projects. Ottawa.

Environment Canada. 2003. Audit of Grants and Contributions Classes 301, 302, 304, 305. Ottawa.

Environment Canada. 2004. Environment Canada's Manager's Guide to Grants and Contributions. Ottawa.

Environment Canada. 2005. Follow-up to the Audit and Evaluation of Class Grants and Contributions. Ottawa.

Environment Canada. 2006. 2006-2007 Estimates: Part III - Report on Plans and Priorities. Ottawa.

Environment Canada. 2007. 2007-2008 Estimates: Part III - Report on Plans and Priorities. Ottawa.

Environment Canada. 2007. Departmental Performance Report, 2006-2007. Ottawa.

Environment Canada. 2007. Towards Improved Management and Accountability of EC's Grants & Contributions[PowerPoint]. Ottawa.

Environment Canada. 2008. Community-based Funding Programs Optimization [PowerPoint]. Ottawa.

Environment Canada. 2008. Departmental Performance Report, 2007-2008. Ottawa.

Environment Canada. 2008. Financial Information related to Class Grant & Contribution Programs for Fiscal Year 2006-2007: Status as of August 19, 2008. Ottawa.

Environment Canada. 2008. Financial Information related to Class Grants & Contributions for Fiscal Year 2007-2008: Status as of August 19, 2008. Ottawa.

Environment Canada. 2008. Grants & Contributions 2008-2009 [PowerPoint]. Ottawa.

Environment Canada. 2008. Grants & Contributions Review Committee - Terms of Reference. Ottawa.

Environment Canada. 2009. Environment Canada's Grants and Contributions [PowerPoint]. Ottawa.

Gardner Pinfold Consulting Economists Limited. 2008. An Update of the Economic Impact of the Atlantic Coastal Action Program (ACAP). Halifax.

Natural Resources Canada. 2007. Results-Based Management Accountability Framework: Departmental Class Grants and Contributions Program. Ottawa.

Treasury Board of Canada Secretariat. 2000. Policy on Transfer Payments. Ottawa.

Treasury Board of Canada Secretariat. 2002. Guide on Grants, Contributions and Other Transfer Payments. Ottawa.

Treasury Board of Canada Secretariat. 2006. From Red Tape to Clear Results: The Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs. Ottawa.

Treasury Board of Canada Secretariat. 2008. Directive on Transfer Payments. Access at: http://publiservice.tbs-sct.gc.ca/pol/doc-eng.aspx?evttoo=X&id=14208&section=text

Annex 3 - Interview Guides

DEPARTMENTAL PROGRAM MANAGERS

Introduction

Environment Canada's Audit and Evaluation Branch is conducting an evaluation of the Department's Class Grants and Contributions (G&C) programs for the 2006-07 and 2007-08 fiscal years. The four class G&C programs at Environment Canada include:

  • Grants to Support Environmental Research and Development
  • Contributions to Support Environmental Research and Development
  • Contributions to Support Regional Environmental and Sustainable Development Projects
  • Contributions to Support Canada's International Commitments

At Environment Canada, class programs may support individual grants and contributions or larger programs which use the class authority for their contributions. A full list of class programs can be found in the attached annex (Annex 1).

As a departmental representative with experience using these class programs, we are seeking your feedback on such things as the ongoing relevance and delivery of the program, as well as the degree to which the classes may contribute to the achievement of departmental objectives. When answering the interview questions, please draw upon your past experience using the class G&Cs and provide specific examples where possible.

Background

To begin, we'd like to get a sense of your area of responsibility within the Department and the nature of your involvement with the class programs.

1. Please briefly describe your program's key areas of activity within Environment Canada (Note: if respondent has changed jobs, focus on previous program areas which made use of the class programs).

2. And what is the nature of your involvement with the class G&Cs programs at Environment Canada?

  • With which particular classes have you been involved? (Please see Annex 1)
  • What types of project activities does your program support through the class programs?
  • Are the class G&Cs used to support agreements that are directly aligned with program activities? Can you provide examples?

Relevance

3. In your view, is there a need for class G&Cs programs to support individuals, organizations or other levels of government in the pursuit of departmental priorities?

  • Is there the same need for support in all areas targeted by the classes (research and development, sustainable development, international commitments)?
  • To what extent do these class G&Cs programs overlap with or duplicate alternative funding sources to support these groups?

4. Thinking of the specific agreements with which you have experience, why was a class program chosen over other funding alternatives?

5. In your view, to what extent do the class G&Cs programs support projects that are consistent with departmental priorities? Please explain.

Design and Delivery

6. To what extent do you feel the class G&Cs programs are being delivered as designed? Please explain.

7. Are you aware of any processes that have been put in place to ensure the proper management and monitoring of the transfer payments at the level of the individual classes? (Prompt: this could include processes related to program delivery, governance and accountability, risk management, performance measurement or reporting).

8. Are the class terms and conditions well suited to the types of projects your division wishes to support?

  • To what extent are funding recipients required to adjust their project designs in order to accommodate these class terms? Please explain.

9. To what extent do funded agreements clearly identify deliverables and expected results/outcomes?

  • Are indicators generally developed and clearly identified for projects supported through the classes?

10. Is performance data for class agreements collected against activities and outcomes? What types of data are available?

  • Is collected information used to inform senior management/decision makers? If so, how is this done?
  • Is this performance information aggregated to demonstrate progress toward specific departmental outcomes? Please explain.

11. Who is accountable for the class G&Cs programs? Do you have a clear sense of the roles and responsibilities of other parties involved in the delivery of the class programs?

  • Are there any factors which impede or enhance accountability for the class programs? If yes, please describe.

Success

12. To what extent have the intended outcomes of funded projects been achieved as a result of the class G&Cs programs? Please describe.

13. And to what degree do you feel funded projects have contributed to furthering departmental policy? Please explain.

14. Have the class G&Cs programs led to any unintended (positive or negative) impacts?

  • Were any actions taken as a result of these unintended outcomes?
  • Have any individual agreements yielded unintended impacts?

Cost-Effectiveness/Alternatives

15. In your view, are there alternative ways of supporting individuals, organizations or other levels of government in the pursuit of departmental priorities?

16. How might the efficiency of class G&Cs programs be improved?

17. What are the advantages and disadvantages of using a class approach over using a specific G&C program approach?

  • From your perspective, do the advantages outweigh the disadvantages?

18. What is the impact of a class approach on evaluation, accountability and reporting?

  • Relative to other types of transfer payment program, are these functions easier or harder to support for class G&Cs programs? Why?

19. What are some of the best practices and key lessons learned from the class G&Cs programs that may be applicable to other similar initiatives?

Conclusion

20. Do you have any additional comments?

Thank You

FINANCE REPRESENTATIVES

Introduction

Environment Canada's Audit and Evaluation Branch is conducting an evaluation of the Department's Class Grants and Contributions (G&Cs) programs for the 2006-07 and 2007-08 fiscal years. The four class G&Cs programs at Environment Canada include:

  • Grants to Support Environmental Research and Development
  • Contributions to Support Environmental Research and Development
  • Contributions to Support Regional Environmental and Sustainable Development Projects
  • Contributions to Support Canada's International Commitments

At Environment Canada, class programs may support individual grants and contributions or larger programs which use the class authority for their contributions. A full list of class programs can be found in the attached annex (Annex 1).

As a member of the Department's financial community with responsibility for managing agreements under these class programs, we are seeking your feedback on such things as the ongoing relevance and delivery of the program, as well as the degree to which the classes may contribute to the achievement of departmental objectives. When answering the interview questions, please draw upon your past experience using the class G&Cs and provide specific examples where possible. You are welcome to invite another member of your team with particular expertise in grants and contributions to participate in the interview if you feel this would be helpful in answering some questions.

Background

1. Please briefly describe the nature of your involvement with the class G&Cs programs at Environment Canada.

  • With which particular classes have you been involved?

Relevance

2. In your view, is there a need for class G&Cs programs to support individuals, organizations or other levels of government in the pursuit of departmental priorities?

  • Is there the same need for support in all areas targeted by the classes (research and development, sustainable development, international commitments)?
  • To what extent do these class G&Cs programs overlap with or duplicate alternative funding sources to support these groups?

3. What do you feel are the main reasons why program managers choose to use class G&Cs over another type of transfer payment program?

4. In your view, to what extent do the class G&Cs programs support projects that are consistent with departmental priorities? Please explain.

5. Are the class G&Cs used to support agreements that are directly aligned with program activities? Can you provide examples?

Design and Delivery

6. To what extent do you feel the class G&Cs programs are being delivered as designed? Please explain.

7. Are you aware of any processes that have been put in place to ensure the proper management and monitoring of the transfer payments at the level of the individual classes? (Prompt: this could include processes related to program delivery, governance and accountability, risk management, performance measurement or reporting).

8. To what extent do funded agreements clearly identify deliverables and expected results/outcomes?

  • Are indicators generally developed and clearly identified for projects supported through the classes?

9. Is performance data for class agreements collected against activities and outcomes? What types of data are available?

  • Is collected information used to inform senior management/decision makers? If so, how is this done?
  • Is this performance information aggregated to demonstrate progress toward specific departmental outcomes? Please explain.

10. Who is accountable for the class G&Cs programs? Are the roles and responsibilities of all groups involved clear?

  • Are there any factors which impede or enhance accountability for the class programs? If yes, please describe.

Success

11. To what extent have class G&Cs overall contributed to furthering departmental policy? Please describe.

12. Have the class G&Cs programs led to any unintended (positive or negative) impacts?

  • Were any actions taken as a result of these unintended outcomes?
  • Have any individual agreements yielded unintended impacts?

Cost-Effectiveness/Alternatives

13. In your view, are there alternative ways of supporting individuals, organizations or other levels of government in the pursuit of departmental priorities?

14. How might the efficiency of class G&Cs programs be improved?

15. What are the advantages and disadvantages of using a class approach over using a specific G&C program approach? Please explain

  • From your perspective, do the advantages outweigh the disadvantages?

16. What is the impact of a class approach on evaluation, accountability and reporting?

  • Relative to other types of transfer payment program, are these functions easier or harder to support for class G&Cs programs? Why?

17. What are some of the best practices and key lessons learned from the class G&Cs programs that may be applicable to other similar initiatives?

Conclusion

18. Do you have any additional comments?

Thank You

RECIPIENTS

Introduction

Environment Canada's Audit and Evaluation Branch is conducting an evaluation of the Department's Class Grants and Contributions (G&Cs) programs for the 2006-07 and 2007-08 fiscal years. The four class G&Cs programs at Environment Canada include:

  • Grants to Support Environmental Research and Development
  • Contributions to Support Environmental Research and Development
  • Contributions to Support Regional Environmental and Sustainable Development Projects
  • Contributions to Support Canada's International Commitments

At Environment Canada, class programs may support individual grants and contributions or larger programs which use the class authority for their contributions. A full list of class programs can be found in the attached annex (Annex 1).

Our records indicate that you have received federal funding from one of Environment Canada's class grants and contributions programs. As such, we are seeking your feedback on such things as the delivery of the program and the degree to which it may have contributed to the success of your project. When answering the interview questions, please draw upon your past experience as a recipient of this funding and provide specific examples where possible.

Background

1. Briefly describe the nature of the project for which you received federal funding from a class grant or contribution.

  • Have you ever used this funding program in the past? If so, please describe.
  • Is your project delivered in partnership with any other individual or group? Please describe.

Relevance

2. In your view, is there a need for federal programs to support projects such as yours? Why?

  • From your experience, does this form of federal funding overlap with or duplicate other funding sources (federal or non-federal)?

Design and Delivery

3. Were you informed at the outset how the funding process would work?

  • What means were used to communicate this information to you?

4. Did the process of obtaining funding through this mechanism unfold as you were informed it would? If not, what was different?

5. Were the terms and conditions for funding well suited to your project?

  • Were you required to adjust the approach to your project in order to accommodate the terms of the funding program? Please explain.

6. To what extent does your project clearly identify deliverables and expected results?

  • Have indicators been developed and clearly identified for your project?
  • If not, why not?

7. Is performance data for your project collected against activities/outcomes?

  • What types of data are available?
  • How is this information communicated to Environment Canada?

8. To what extent do you clearly understand your roles and responsibilities with respect to this funding mechanism, as specified by the terms of your Contribution Agreement?

  • Do you have a clear sense of the roles and responsibilities of other parties involved in the delivery of this federal funding mechanism?
  • What is your understanding of EC's role?

Success

9. To what extent has your project achieved its intended outcomes? Please describe.

  • Would these results have been achieved if funding wasn't provided through this mechanism?

10. Has your project had any unintended (positive or negative) impacts? If yes, please describe and provide examples.

  • Were any actions taken as a result of these unintended outcomes?

Cost-Effectiveness/Alternatives

11. Are there alternative ways of achieving your project's objectives?

12. Thinking of such things as the Contribution Agreement or the manner in which funds are accessed (e.g., administrative processes), how might the efficiency of this funding program be improved? Please describe.

13. Overall, what do you feel are the strengths and weaknesses of this funding mechanism?

Conclusion

14. Do you have any additional comments?

Thank You

Annex 4 - Summary of Findings

Evaluation Question (EQ)AchievedProgress Made, Attention NeededLittle Progress,Priority for AttentionNot Applicable
Relevance:
EQ1 Need for Class G&CsX   
EQ2 Consistency with departmental prioritiesX   
Success:
EQ3 Achievement of project outcomesX   
EQ3 Achievement of departmental objectives X  
EQ4 Unintended outcomes   X
Cost Effectiveness:
EQ5 Alternatives/improved efficiency X  
EQ6 Value for federal dollarsX   
EQ7 Advantages and disadvantages   X
EQ7 Evaluation, accountability and reporting  X 
Design and Delivery:
EQ8 Identification of deliverables and resultsX   
EQ9 Delivered as designed X  
EQ10 Performance measurement and reporting X  
EQ11 Accountability X  
EQ12 Best practices and lessons learned   X

[1] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa.

[2] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa.

[3] Descriptions of all four classes are adapted from their respective terms and conditions, as reproduced in annexes B, C, D and E of Environment Canada. 2004. Environment Canada's Manager's Guide to Grants and Contributions. Ottawa.

[4] Environment Canada. 2007., 2006-07 Departmental Performance Report. Ottawa. p. 16.

[5] Environment Canada. 2008., 2007-08 Departmental Performance Report, Ottawa. Table 2, p. 56.

[6] Environment Canada. 2007.Presentation to SI Board, Towards Improved Management and Accountability of EC's Grants and Contributions [PowerPoint]. Ottawa., April 4, 2007.

[7] Environment Canada. 2007.Presentation to SI Board, Towards Improved Management and Accountability of EC's Grants and Contributions [PowerPoint]. Ottawa.

[8] Ibid.

[9] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa.

[10] The numbers of strategic and ultimate outcomes, OPGs and OPPs presented in this logic model are based on a 2007-08 version of the Department's Program Activity Architecture.

[11] Two Youth Employment Strategy programs (Science Horizons and International Environmental Youth Corps) were not included in the sample frame for the file review, as these two programs are no longer part of the departmental classes and were evaluated separately in 2008-09 as part of their own terms and conditions.

[12] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. Appendix G.

[13] Treasury Board of Canada Secretariat. 2008., Directive on Transfer Payments. Access at: http://publiservice.tbs-sct.gc.ca/pol/doc-eng.aspx?evttoo=X&id=14208&section=text, sections 6.3.1 and 6.8.2, October 1, 2008.

[14] Treasury Board of Canada Secretariat. 2002., Guide on Grants, Contributions and Other Transfer Payments. Ottawa. Section 8.5.2.

[15] Environment Canada. 2001. Audit and Evaluation of Class Contributions in Support of Environmental and Sustainable Development Projects. Ottawa; Environment Canada. 2003. Audit of Grants and Contributions: Classes 301, 302, 304, and 305. Ottawa; Environment Canada. 2005. Follow-up to the Audit and Evaluation of Class Grants and Contributions. Ottawa.

[16] During the period in question for the current evaluation, departmental activities were organized using a results management structure (RMS) divided into: Outcome Project Sub Components (OPSCs), the smallest element of the results structure where short-term results related to work planning are articulated; Outcome Project Plans (OPPs), which embodied 169 medium-term results statements to which OPSCs were aligned; and 39 higher-order outcomes represented by Outcome Project Groups (OPGs) comprised of groupings of OPPs. All levels of the RMS were aligned with the Department's three strategic outcomes.

[17] Treasury Board of Canada Secretariat. 2006. From Red Tape to Clear Results: The Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs. Ottawa.

[18] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. pp. 4-5.

[19] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. p. 80.

[20] Some program and finance respondents noted incorrectly that Contributions to Support Canada's International Commitments may no longer be required under the new Transfer Payment Policy because these dues can now be paid through contracts. Contracts cannot be used in this fashion and the new Transfer Payment Policy which came into effect on October 1, 2008, allows that transfer payments can include payments to other orders of government, international organizations and Aboriginal peoples (Section 3.1).

[21] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa.

[22] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. p. 78.

[23] Environment Canada. 2007. Towards Improved Management and Accountability of EC's Grants and Contributions [PowerPoint]. Ottawa.

[24] Environment Canada. 2007. Towards Improved Management and Accountability of EC's Grants and Contributions [PowerPoint]. Ottawa. slide 12.

[25] Other outcomes include those related to the Internal Services Board and the now-defunct Climate Change Board.

[26] For example, agreement files that included a Grant and Contribution Approval Form provide a descriptive summary of the agreement, as well as a discussion of the Recipient Eligibility, Project Eligibility and Purpose, Project Priority and Risk Assessment.

[27] Environment Canada. 2005. Follow-up to the Audit and Evaluation of Class Grants and Contributions. Ottawa.

[28] Treasury Board of Canada Secretariat. 2000. Policy on Transfer Payments. Ottawa.

[29] These files contained the following types of reporting: final report (42%); progress, interim or activity reporting (38%); or an annual report (17%).

[30] The nature of outputs varies substantially according to the objectives and activities of the agreements, and include a range of outputs such as research studies and data collection, product development (e.g., Web sites, planning frameworks), events/consultations/ professional development, and other outputs (e.g., infrastructure upgrades/maintenance, Canadian membership to international organizations/funds).

[31] This proportion is substantially higher than the percentage of files containing evaluation-related studies (15%) because descriptive evidence of outcomes (e.g., included in final reports, annual reports, correspondence) was included here as "evidence of outcomes" even though this evidence is not strictly derived from evaluative evidence.

[32] It is important to point out, however, that the new Transfer Payment Policy (October 1, 2008) was not in effect during the timelines examined for the current evaluation.

[33] Environment Canada. 2007. 2006-07 Departmental Performance Report. Ottawa. p. 16.

[34] Environment Canada. 2008. 2007-08 Departmental Performance Report. Ottawa. Table 2, p. 56.

[35] The class G&C spending in 2006-07 was significantly lower than in 2007-08, with the difference largely accounted for by the sunsetting of a number of temporary programs funded through vote netted revenues, including CoP-11 ($7M), the Opportunities Envelope ($4.7M), the One-Tonne Challenge ($3M), and the Partnership Fund ($5M). Furthermore, EC's Expenditure Restraint Reductions reduced overall departmental G&C spending by $1.7M in 2006-07 and $5.9M in 2007-08.

[36] (($92,676,880 - $14,207,184)/$14,207,184) = 5.5.

[37] Environment Canada. 2004., Manager's Guide to Grants and Contributions. Ottawa. p. 76.

[38] Although percentages related to matching funds are not specified, a stacking provision discussed in Environment Canada's Manager's Guide to Grants and Contributions states that the maximum level (stacking limit) of Total Government Assistance (federal, provincial and municipal assistance for the same eligible expenditures) should not exceed 100% of eligible expenditures (p. 34).

[39] The Atlantic Coastal Action Program (ACAP) is a community-based model of ecosystem management. There are 14 ACAP organizations throughout Atlantic Canada working toward their collective goal of developing local capacity to take responsibility for their own futures. The ACAP is funded through the Ecosystems Initiative sub-class G&C program, which is delivered under the Contributions to Support Regional Environmental and Sustainable Development Projects class authority (G03).

[40] Gardner Pinfold Consulting Economists Limited. 2008., An Update of the Economic Impact of the Atlantic Coastal Action Program (ACAP). Halifax. p 7.

[41] Treasury Board of Canada Secretariat. 2006. From Red Tape to Clear Results: The Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs. Ottawa. p. vii.

[42] Natural Resources Canada. 2007. Results-Based Management and Accountability Framework: Departmental Class Grants and Contributions Program. Ottawa, p 3.

[43] It should be noted, however, that Environment Canada's Manager's Guide to Grants and Contributions clearly distinguishes when a program manager should use a contract versus a contribution agreement (p. 6).

[44] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. p. 71.

[45] The contribution agreement templates states that "Any intellectual property rights arising from the project will be vested with the Recipient provided that the Recipient hereby grants to the Minister the licensed rights to produce, publish, translate, reproduce, adapt, broadcast or use at no cost any work subject to such intellectual property rights."

[46] One respondent also suggested that it would be helpful for the programs to develop marketing material to help potential recipients access other resources and partners.

[47] A new Policy on Transfer Payments came into effect on October 1, 2008. The current study evaluated the class G&Cs in the context of the previous Policy on Transfer Payments which was in effect during the timeframe in question.

[48] Treasury Board of Canada Secretariat. 2000. Policy on Transfer Payments. Ottawa. Section 8.3.1.

[49] Environment Canada. 2005.Follow-up to the Audit and Evaluation of Class Grants and Contributions. Ottawa.

[50] The G&C angiogram exercise involved consultations with various stakeholders across the Department, as well as a review of contribution agreement files to assess the time required to approve, negotiate and sign contribution agreements.

[51] Only 3% of files reviewed for this evaluation contained information related to formal audits or evaluations of the recipient's project.

[52] Treasury Board of Canada Secretariat. 2000. Policy on Transfer Payments. Ottawa. Section 7.11.5.

[53] Environment Canada. 2005. Follow-up to the Audit and Evaluation of Class Grants and Contributions. Ottawa. pp. 19 and 20.

[54] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. Annexes B, C, D, E and G.

[55] Ibid. Appendix K, p. 79.

[56] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. Annex F.

[57] Treasury Board of Canada Secretariat. 2000. Policy on Transfer Payments. Ottawa. Section 7.11.6.

[58] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. Contribution Agreement Template.

[59] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. Appendix K.

[60] Environment Canada. June 2004. Environment Canada's, Manager's Guide to Grants and Contributions. Ottawa. Appendix K, p. 83.

[61] Environment Canada. 2005. Follow-up to the Audit and Evaluation of Class Grants and Contributions. Ottawa.

[62] Treasury Board of Canada Secretariat. 2000., Policy on Transfer Payments. Ottawa. Section 7.4.7.

[63] Treasury Board of Canada Secretariat. 2000., Policy on Transfer Payments. Ottawa. Section 7.1.

[64] The ARAF focuses on the previous business line matrix management accountability structure of the Department and does not reflect the current departmental PAA.

[65] Environment Canada. 2007. Towards Improved Management and Accountability of EC's Grants and Contributions [PowerPoint]. Ottawa.

[66] Environment Canada. 2008. Grants & Contributions 2008-2009 [PowerPoint]. Ottawa.

Date modified: