Evaluation of the Co-location of Science Research Centres on University Campuses

2.1    Background

Collaborative Arrangements

Environment Canada (EC) is involved in a wide range of collaborative arrangements with its science and technology (S&T) partners.  These range from student employment, through appointment of staff as adjunct professors at universities, research chairs/partnerships, networks and joint publications, to facility and equipment sharing. [1]  The goal of all of these forms of collaborative arrangements is to increase capacity to deliver on the Department’s mandate and on government priorities. [2]   EC’s Smart Partners working paper states that “through S&T partnerships, the Department builds synergy with other organizations, levers resources, enhances human resource development, promotes the use of research and development (R&D) results, and draws on S&T expertise in other sectors.” [3]

In the March 2000 report, Building Excellence in Science and Technology: The Federal Roles in Performing Science and Technology, the Council of Science and Technology Advisors (CSTA) recommended the implementation and funding of new models for S&T that move away from a vertical approach (single organization) to a horizontal multi-stakeholder approach.  An example of this multi-stakeholder approach would be building connections between universities and federal science-based departments.  This recommendation reinforces the innovative approaches EC is undertaking through a variety of collaborative arrangements.

The most recent CSTA report (2005), Linkages in the National Knowledge System: Fostering a Linked Federal S&T Enterprise, calls for the government to explore options for co-location of S&T facilities, both among government departments and between government and academic or industrial organizations. [4]

Co-location

Within EC, one type of collaborative arrangement the department employs is co-location.  This includes the location of individual scientists or facilities on a university campus, where opportunities for partnerships are improved simply by physical proximity and the ease with which connections can be made.  Co-location arrangements are described as “offer[ing] increased opportunities for research collaboration and other shared activities, access to facilities, and to students at the local level.  Through these contacts greater connections with the broader academic community are possible.” [5] 

This evaluation focuses specifically on the co-location of EC research centres on university campuses.  As noted in Smart Partners: Innovations in Environment Canada-University Research Relationships, EC recognizes that partnerships and networks with academic researchers are of joint benefit.  Working closely with university colleagues helps to keep EC scientists up-to-date with current scientific thinking and also provides them with direct access to graduate students.  Universities also benefit from increased research opportunities, teaching and research supervision for their students, as well as access to EC’s unique facilities and specialized equipment. [6]

EC’s Policy on the Approval and Management of Collaborative R&D/Teaching Positions for S&T Professional Employed by EC states that “EC encourages it’s S&T professional staff to seek and accept appointments in collaborative R&D/teaching positions as a means of collaborating with other agencies, aiding in the training of students, and furthering the department’s R&D in priority areas.”  The policy goes on to define a collaborative position as being “a post, position or appointment with an organization outside the Government of Canada in which the individual collaborates with the staff and/or students of that organization toward a common end which will benefit the organization and the Government of Canada”. [7]

The department recognizes the advantages of collaborative positions for developing an integrated approach to environmental science. Thus, the most recent count revealed that at least 200 EC employees held collaborative positions in 30 universities acrossCanada. [8]

2.2    Logic Model

The logic model on the following page is a graphic depiction of co-location that describes the progression from inputs and activities through to outcomes. [9]  This logic model was developed by the Audit and Evaluation Branch as a generic representation of the co-location process. This was compiled by amalgamating identified benefits of co-location from the following documents:

• Manager’s Guide: Environment Canada Policy and Approvals: Collaborative Positions for S&T Professionals, Environment Canada , January 1999;
• Current Partnering Policy and Practice at Environment Canada , August 2000;
• Working with Others: Policy on Revenue and Collaborative Arrangements, EC, December 2000; and
• Smart Partners: Innovations in Environment Canada-University Research Relationships, 2004.

The logic model was developed as a framework by which examples of EC’s co-location arrangements could be assessed.   For the purpose of this evaluation the framework was used to examine two specific case studies, however, it could also be used to examine other examples of co-location.  In order to assess the case studies against the outcomes identified in the logic model it is critical to keep in mind the specific context under which each example of co-location took place.

Figure 1:  Logic Model for the Co-location of Science Research Centres on University Campuses

2.3    Case Studies

For the purpose of this evaluation, two examples of co-location on university campuses were selected by the Departmental Audit and Evaluation Committee (DAEC) as case studies; these were the Canadian Centre for Climate Modelling & Analysis (CCCma) and the National Wildlife Research Centre (NWRC).   The case studies are not meant to represent all of the EC co-location examples but rather provide two informative cases from which to learn. 

The case studies vary considerably from one another and while they were both assessed against the same criteria (logic model) it is important to note that they had different reasons for co-locating, have different space and facilities needs, and have operated for different periods of time under diverse mandates.   

Canadian Centre for Climate Modelling & Analysis (CCCma)

CCCma is a division of the Climate Research Branch of the Meteorological Service of Canada of EC. [10]  They conduct research in coupled and atmospheric climate modelling, sea-ice modelling, climate variability and predictability, the carbon cycle, and a number of other areas.

The following is the stated key result of the Centre - Global and Regional Climate Modelling:

To understand and predict states of the atmosphere, hydrosphere and cryosphere on all time and spatial scales to meet client and decision-makers’ needs.[11]

The CCCma is the focal point in Canada for the development and operation of climate models.  The group, consisting of approximately 34 scientists, support staff, research associates and post-doctoral fellows (approximately 23 full-time EC staff supplemented by students and DFO secondments), works closely with colleagues from several Canadian universities and the Department of Fisheries and Oceans. [12]  Of the 34 staff almost half are research scientists.

CCCma was moved from Downsview , Ontario to their present location on the University of Victoria (UVic) campus in 1993.  At the time of the move both the east ( Halifax ) and west ( Victoria ) coasts were considered.  The reason for the move was to establish close collaboration with ocean modellers in the UVic School of Earth and Ocean Science and the Department of Fisheries and Oceans’ Institute for Ocean Sciences, also located near Victoria . 

The CCCma is currently working with UVic to move their centre, currently located in the Ian Stewart Complex, to the new Science Building being constructed on campus.  The CCCma is likely to take occupancy in 2008/09. [13]

National Wildlife Research Centre (NWRC) [14]

The National Wildlife Research Centre is the focal point within the Canadian Wildlife Service [15] for:

• Research and advice on the effects of toxic substances on wildlife (Wildlife Toxicology Division); and
• National research and surveys on migratory birds (Migratory Bird Populations Division). [16]

The following are the two stated key results the Centre strives to achieve - Migratory Birds Monitoring & Research: national level science is provided to support the conservation of migratory birds; and Wildlife Toxicology: impacts of toxic substances on health of wildlife and their habitat are understood and reduced.  Wildlife are used as indicators of environmental quality and ecosystem health as well as early warning indicators of human health problems. [17] 

The mission of the NWRC is to be the principal source of knowledge and expertise in the federal government on the impact of toxic substances on wildlife and the use of wildlife as indicators of environmental quality, and to conduct national surveys and research on migratory birds. [18]

The Centre, with a staff of more than 75 employees (approximately 60 permanent EC staff, of which one quarter are research scientists) including contractors and students, moved from an aging research building in Hull, Quebec in November 2002 to new research facilities on the Carleton University campus in Ottawa, Ontario.  

The decision to move stemmed from health and safety concerns due to aging facilities.  An assessment of the alternatives was conducted comparing the Centre’s various options, such as rebuilding at the Gamelin site, co-locating with another EC facility in Ottawa , or co-locating on a university campus. The rating completed through the qualitative and quantitative analyses revealed that co-locating with a University was the best option. As a result, both the University of Ottawa and Carleton University submitted proposals to accommodate the Centre.  During the assessment process the University of Ottawa withdrew their proposal and Carleton University was selected to be the site of the NWRC. 

2.4    Purpose and Scope of the Evaluation [19]

In order to conduct an evaluation that assesses the impact of co-locating EC’s scientific research centres on university campuses with a focus on:

1. the forecast benefits and synergies of co-location (e.g., with the university and wider community) versus those actually realized;
2. the extent to which the science undertakings and results support departmental strategic outcomes and have the flexibility to respond to changing needs; and
3. an examination of cost-effectiveness with regard to the decision to co-locate and annual operating costs (e.g., leases, agreements).

Both the NWRC at Carleton University and the CCCma at the University of Victoria were used as co-location case studies in this evaluation. This is an evaluation of co-location using case studies rather than an evaluation of the centres, their programs and activities.

2.5    Key Issues and Evaluation Questions

The evaluation plan identified that the evaluation wouldexamine the following four evaluation issues in order to address the purpose of the evaluation, identified in the previous section:

a)     The issue of relevance assessed whether co-location addresses actual needs;

b)     The issue of success focused on whether co-location was on track to meeting its intended outcomes;

c)     The issue of cost-effectiveness investigated whether the most appropriate and efficient means were being used to achieve outcomes; and

d)     Finally, the issue of design and delivery investigated the extent to which co-location is being designed and delivered in the best possible way.

The evidence for this evaluation was collected between September 2005 and February 2006.The specific evaluation questions pertaining to each evaluation issue are detailed in Annex 1.

2.6    Methodology

In accordance with best practices[20], the approach for the evaluation involved the use of multiple methods of analysis detailed below.  

Document and File Review

Policy and planning documents as well as historic documentation on the co-location of the research centres were reviewed. In addition, documentation on specific outcomes (e.g., number of students, adjuncts, publications, resource levels) was assessed.  Finally, other types and examples of collaborative arrangements were researched.  Document review was an integral component in this evaluation; a full list of the documents reviewed can be found in Annex 2.  Examples of documents that were collected included: mandates, leases, financial reports, and Treasury Board Submissions. 

Given that document review is a key source in the evaluation, it should be noted that a general lack of documentation and records posed a challenge to the evaluation.  Limited documentation was available upon request by evaluators; the remainder of the documentation was either not being used and therefore not readily available or did not exist.  In addition, documentation pertaining to financial information contained inconsistencies with regard to the terms and definitions used.  This made comparability of documentation and the overall assessment of cost-effectiveness an issue. 

Interviews

Interviews with key stakeholders were conducted for the purpose of the evaluation.  Annex 3 provides a complete list of all of the interviews conducted.

For each of the two case studies, interviews were divided into a number of groups: EC centre management and staff (including adjuncts, researchers); EC senior management (non-centre); EC policy staff; and university faculty.  Sixteen interviews were conducted for the CCCma case study and another fifteen interviews were conducted for the NWRC case study. [21]  Interviewees were suggested by the evaluation committee that was formed to guide the evaluation. 

International experts in climate modelling were also surveyed for their opinions on the CCCma and the impact co-location has had on the Centre.  Experts were identified by centre management and ten experts received an email asking them to respond to five questions; four responses were received.  Due to the national nature of the NWRC’s work and the fact that they focus on multiple areas of research, experts were not surveyed for their opinion on this case study.

Finally, interviews were conducted with key staff from a variety other examples of co-location/collaboration.  Suggestions for examples of other instances of co-location were gathered throughout the interviews as well as through internet research.  In total, ten examples were identified and six interviews were conducted in the allotted time frame.

Interviewees were contacted in advance to inform them of the purpose of the interview and request their participation.  Interview questions were provided by email prior to the interviews.  See Annex 4 for a complete list of interview questions by group.

Cost-Effectiveness Analysis

Cost-effectiveness analysis was identified as a method which would provide comparative information on the costs associated with relocating as well as information on alternative partnering options.

In the case of the CCCma case study, there was insufficient information to conduct this type of analysis as little documented information exists on the move in 1993 and no comparative documentation of alternative locations considered was identified.  Therefore, no cost-effectiveness analysis could be conducted for this case study.

For the NWRC case study, a cost comparison of locations for relocating was completed.  However, there is little consolidated documentation to identify post-co-location costs and whether these were in line with original projections.  As such, no cost-effectiveness analysis could be conducted for this case study.

Bibliometric Analysis

Bibliometric analysis was identified in the plan as a method that would provide comparative information on the case studies in areas such as publications (pre/post co-location).  In pursuing this method, which was to be contracted out, consultants who specialize in this area suggested that the sample size for the research centres was insufficient to provide reliable results.  Given this information, bibliometric analysis was not conducted as part of the evaluation.  Information about publications was subsequently collected through interviews and document review.