May 2006

Report Clearance Steps

Acronyms used in the report

Acknowledgments

The Audit and Evaluation Project Team including V. Neimanis, Anne-Marie Bedard and led by Shelley Tice under the direction of the Director, Evaluation Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights, and comments crucial to this evaluation.

Report Prepared by the Evaluation Division, Audit and Evaluation Branch

EXECUTIVE SUMMARY

As identified in the Departmental Audit and Evaluation Plan 2005/6 to 2007/8, an evaluation of the co-location of science research centres on university campuses was initiated because of potential risks associated with a relatively new delivery mechanism (co-location) and to ensure research centres continue to align with departmental needs.

This evaluation assessed the impact of co-locating Environment Canada’s (EC) scientific research centres on university campuses with a focus on:

1. the forecast benefits and synergies of co-location (e.g., with the university and wider community) versus those actually realized;
2. the extent to which the science undertakings and results support departmental strategic outcomes and have the flexibility to respond to changing needs; and
3. an examination of cost-effectiveness with regard to the decision to co-locate and annual operating costs (e.g., leases, agreements).

The National Wildlife Research Centre (NWRC) at Carleton University and the Canadian Centre for Climate Modelling & Analysis (CCCma) at the University of Victoria were used as co-location case studies in this evaluation.  The case studies are not meant to represent all of the EC co-location examples but rather provide two informative cases from which to learn. 

The case studies vary considerably from one another and while they were both assessed against the same criteria it is important to note that they had different reasons for co-locating, have different space and facilities needs, and operate under very different mandates.  

In order to assess the case studies, a logic model was developed which identified the predicted outcomes of co-location.  For the purpose of this evaluation, the framework was used to assess two specific case studies; however, it could also be used as a framework to assess other examples of co-location in the future.

The evaluation examined the following four evaluation issues:

a)     The issue of relevance assessed whether co-location addresses actual needs;

b)     The issue of success focused on whether co-location was on track to meeting its intended outcomes;

c)     The issue of cost-effectiveness investigated whether the most appropriate and efficient means were being used to achieve outcomes; and

d)     Finally, the issue of design and delivery investigated the extent to which co-location is being designed and delivered in the best possible way.

In accordance with best practices, the approach for the evaluation involved the use of multiple methods including document review, interviews, and cost-effectiveness analysis.

The following represent the summary findings from this report by evaluation issue.

Evaluation Issue: Relevance

Beyond outcome project plans and mandates which are by their nature expected to align to departmental outcomes, there is limited information to determine the degree to which alignment to departmental strategic outcomes is present.

Evaluation Issue: Success

The reporting of the success issue is according to the three levels of outcomes [1] (immediate, intermediate, final) as well as unintended outcomes.

a) Immediate Outcomes

In the case of the CCCma, there is evidence of the achievement of all of the immediate outcomes (increased access to students, knowledge, specialized facilities, and increased leveraging of resources).  In the case of NWRC, there is limited evidence and achievement of the following outcomes: access to students, knowledge and specialized facilities; the outcome on increased leveraging of resources was not achieved.

b) Intermediate Outcomes

For the CCCma, there is evidence of the achievement of intermediate outcomes (increased synergy and scientific capacity).  For the NWRC, there is limited evidence and achievement of the intermediate outcomes.

c) Final Outcomes

There is evidence of the achievement of the final outcome (scientific research that supports departmental strategic and intermediate outcomes and OPG results) for the CCCma and limited evidence and achievement for the NWRC, however the achievement of this outcome appears to be un-related to co-location.

d) Unintended Outcomes – Centre specific

A number of unintended outcomes were identified that were specific to the individual case studies. 

Evaluation Issue: Cost-Effectiveness

There is insufficient data and information to determine whether or not co-location was cost-effective.

Evaluation Issue: Design and Delivery

The reporting of the design and delivery issue is separated into two areas: alternative designs for collaborative arrangements and best practices/lessons learned.

a) Alternative Designs for Collaborative Arrangements

A number of factors influence the impacts of co-location, includingtype of agreement (between university and federal department), building occupancy, and staff size.

b) Best Practices/Lessons-learned

A number of lessons learned were gathered that can be grouped by two broad themes: facilities and building partnerships.

In conclusion, this evaluation should be useful in further thinking about the benefits of co-location, particularly in the context of discussions to develop the new Science and Technology Plan, due for completion in fall 2006. 

The case studies and information gathered from other examples of co-location demonstrate that there can be clear benefits to co-location.  These benefits vary considerably by individual case however, and are directly linked to the reasons for co-location.

There are a number of different reasons for co-location which impact the level to which the outcomes of co-location are achieved.  It is therefore, critical to look at co-location in terms of the overall context of the partnership.  The most common reasons for co-location include relocating to enhance a scientific knowledge base and build synergies with a wider community (partnership focus), relocating to seek financial or cost-sharing benefits that may result from being co-located (financial focus),  and relocating to build or replace a scientific facility (facility focus).  In certain cases all of these reasons for relocation are present while in others there is one dominant reason. 

Given the findings that were identified by evaluating the two case studies it is possible that improvements could be made at both centres (e.g., enhancing/developing the relationship between the Centre and the host department) that would enhance their co-location outcomes.  These decisions, however, are not limited solely to improving the impacts of co-location; they are also dependent on related management decisions and the overall context for the centres’ operations.  Furthermore, the information learned through this evaluation would be useful in assessing other existing co-location arrangements as well as future co-location opportunities.

The following are the two recommendations that resulted from the evaluation:

1. The ADM, Science and Technology, should review, by December 2006, the findings for each of the case studies (CCCma and NWRC) and assess whether actions are necessary to improve their co-location outcomes.
2. In the future, when considering the function of existing co-location arrangements or creating new ones, the ADM, Science and Technology should:

a.     Consider other centres that are co-located on university campuses that were not included in this evaluation and use this evaluation as a framework to assess those centres to determine the impact of co-location;

b.     Apply the lessons learned from the case studies to improve future co-location decisions; and

c.     Ensure that the general needs for performance measurement/documentation are met in order to be able to assess co-location and determine its impacts.  It should be noted that the ADM, Finance and Corporate is also responsible for documentation as Assets, Contracting and Environmental Management falls under this responsibility and plays a key role in the physical relocation of facilities.

Management Response: Recommendation 1

The ADM, Science and Technology commits to review, by December 2006, the findings for each of the case studies and assess whether actions are necessary to improve their co-location outcomes.

Management Response: Recommendation 2

In the future, when considering the function of existing co-location arrangements or creating new ones, the ADM, Science and Technology, will use this evaluation as a framework to assess the impact of co-location and to apply the lessons learned from these case studies.

The ADM, Science and Technology, jointly with the ADM, Finance and Corporate, will review the general needs for performance measurement/documentation and communicate the results of this review to Branch managers in order to ensure that these needs are met and that the effectiveness and impact of future co-locations can be assessed.