This report presents the results of the Evaluation of Research and Development, Production and Monitoring in Support of Weather Prediction conducted by Environment Canada’s (EC’s) Audit and Evaluation Branch between March and October 2010.
The evaluation was designed to address standard evaluation issues pertaining to program relevance and performance, as well as senior management information requirements regarding the effectiveness of decision making related to departmental scientific activity in support of weather predictions. The period examined is from 2007-08 to 2009-10.
The Meteorological Service of Canada (MSC) is the organizational branch within EC with responsibility for producing and disseminating weather and environmental services, including those related to weather prediction. The MSC relies on the Atmospheric Science and Technology Directorate (ASTD) of the Science and Technology Branch (S&TB) as a key contributor to meteorological research and development, and on the Major Projects and Supercomputing Directorate of the Chief Information Officer Branch (CIOB) for key activities related to information management and information technology (IM/IT).1
For the purpose of this evaluation, MSC’s work related to climate change was excluded from the scope of the evaluation. Although related to weather, climate change was excluded because it involves work on much longer time scales. Other areas such as air quality, seasonal prediction, and ice monitoring were included in the analyses conducted for this evaluation, as they are integral to and very closely interrelated with weather prediction, and it is not possible to easily extract them from weather prediction activities.
The intended focus of this evaluation is on "science" activities in support of weather prediction. As such, this evaluation does not cover a specific program or group of programs. Rather, it targets a cross-cutting function (science) that involves several organizational units within the Department in support of one category of services, weather prediction, delivered by the MSC.
Following consultation with departmental managers, it was acknowledged that "science" comprises several different activities, including research and development, production and monitoring (hence the title for this evaluation). For the purpose of this evaluation, "science" was classified in two main categories:
The evaluation methodology included a review of relevant federal and departmental documents, as well as key informant interviews with a range of program stakeholders.
Findings and Conclusions
This evaluation of science in support of weather prediction showed that there is a need and a legitimate role for EC to conduct scientific activities in support of weather prediction. Weather prediction is a science-based activity relying on science research and continuous improvements as essential functions of the weather production innovation chain. Such science is essential to support the provision of accurate and timely weather information that enables other government service providers and Canadians to make informed decisions that protect their health and safety against changing weather conditions. The science also contributes to meeting EC’s departmental mandate to provide meteorological services and to supporting the mandates of several federal stakeholders. As well, science is needed by private sector stakeholders to ensure the safety and efficiency of their operations. Finally, active involvement in cutting-edge research is required for the MSC to uphold its international obligations as a member of the World Meteorological Organization (WMO) and to maintain its highly interdependent collaborative relationships with other countries’ meteorological services. In light of these findings, science activities in support of weather prediction are relevant to current needs and government priorities. Furthermore, future science innovations are likely to remain relevant and necessary in order to address evolving client demands, which are in part prompted by science innovations being developed not only within MSC but also by other countries worldwide.
The performance of the science activities in support of weather prediction was measured in terms of the extent to which its end users’ current needs are being met, the effectiveness of its science knowledge transfer along the weather prediction innovation chain, and the adequacy and effectiveness of its priority‑setting and decision‑making mechanisms.
In order to deliver improved services to external clients, those involved in the delivery of weather prediction need to be part of an integrated innovation chain, supported by the appropriate infrastructure and processes and collaboration from the global meteorological community. The needs of external clients of weather prediction are diverse, expanding and becoming increasingly sophisticated as a result of increasing possibilities due to evolving science and technology.
Evidence showed that EC’s weather prediction service is generally viewed as providing high‑quality information and products, and as relying on a wealth of knowledge and expertise. EC is also recognized among the international meteorological community as a valued contributor to scientific collaboration and a world leader in a number of key areas. EC’s traditional weather services are working well and continue to improve. However, evidence indicates that more could be done to enhance EC’s weather prediction products and the mechanisms of delivery, to increase access by, and better support the evolving decision-making needs of, external clients.
Key literature highlights the importance of a science‑based function with "a collaborative research approach spanning the entire [knowledge to action] process" whereby "[r]esearchers and decision makers are engaged together from initially identifying the research question through to applying the knowledge."2 Departmental stakeholders involved in the delivery of weather predictions clearly identified with being part of an innovation chain, and recognized the benefits of collaboration both with internal colleagues and with the broader science community domestically and internationally. Internally, the interaction between research, development, production and monitoring functions appears strong, but more could be done to strengthen communications between these roles and the service function in order to ensure that the external client perspective is adequately considered in science decisions, and to ensure that the service function can have an improved understanding of emerging science to help anticipate what may be possible in the future. A gap was also identified in the area of transferring science developed in the MSC’s national laboratories into operations, and in ensuring that information is shared in a manner that allows it to effectively support the decision-making needs of the Department’s diverse group of clients. These findings indicate that, in order to maximize the effectiveness of weather prediction activities, some improvements are needed to the transfer of science knowledge between those involved in developing such knowledge and those primarily tasked with transferring end products to weather prediction clients.
Several mechanisms are in place for weather prediction priority setting and decision making. These mechanisms appear to generally function well, and recent improvements such as the creation of the Meteorological and Environmental Predictions Innovation Committee (MEPIC) (co-chaired by Directors General [DGs] from the MSC and S&TB), CIOB representation on the Weather and Environmental Services (WES) Board3,and the development of signature projects to pursue senior management’s vision for the future of the MSC, are appropriate means to address ongoing needs for improved coordination and for integration of the S&TB and CIOB support functions in the overall weather prediction science priority‑setting and decision‑making process. The need remains, however, for more extensive communication of senior management’s vision and priorities for allocation of limited departmental resources to those involved in the delivery of weather predictions.
Available evidence suggests that the level of effort expended for weather prediction activities, including science, yields commensurate or better value for given resources and that the delivery models adopted are appropriate. The automation of weather forecasts was highlighted as a key change introduced by the MSC that has resulted in some increased efficiencies. Furthermore, the recent introduction of the Quality Management System (QMS) shows early evidence of benefits that will likely, in the near future, outweigh the investment of time and resources required to implement and maintain the System. Benefits observed to date include articulation of roles and responsibilities, mapping of client/supplier relationships and dependencies, documentation of processes, and reporting on performance. These products are important tools to support efficient and effective decision making and knowledge transfer. Also, the QMS approach contributes overall to promoting a performance measurement culture, which is a key contributor to continuous improvement. This evaluation evidence leads to the conclusion that the implementation of the QMS is beneficial for the organization. Also, there were no clear opportunities for efficiency improvements or alternative modes of delivery for EC’s weather prediction science identified as part of this evaluation.
The effectiveness, efficiency and sustainability of weather prediction activities are, however, influenced by a number of external factors. Serious concerns were expressed by departmental stakeholders and confirmed by external reviews (e.g., the Commissioner of the Environment and Sustainable Development’s 2008 audit of severe weather warnings) with regards to the Department’s limited capacity to maintain and upgrade its rapidly aging monitoring infrastructure, and its supercomputer’s limited capacity to meet the demands of all internal and external clients. These factors affect the ability of weather prediction science to respond to weather prediction client needs. These observations lead to the conclusion that the MSC’s ability to continue to be responsive to client needs and to maintain its current level of service are largely dependent on factors that are outside of its direct control.
Three main, interrelated areas for improvement emerge from the evidence gathered as part of this evaluation. First, in light of evolving demands for weather prediction products and services and the limited availability of resources, the Department should identify priorities in terms of client groups to serve and specific client needs to address (i.e., specific products to develop and levels of services to offer). Such strategic choices will have a significant influence on the type of science activities pursued by the research, development, production and monitoring functions.
Second, the identification of departmental priorities should be done in consultation with members of all functions along the weather prediction innovation chain. Priority setting should consider not only the external clients’ needs but also the need to remain at the cutting edge of scientific research in order to maintain Canada’s strategic positioning in the top tier group of countries providing meteorological services, and in order to foster the innovation needed to remain relevant to end users.
Third, the priorities, once established, should be clearly communicated throughout the innovation chain in order to effectively support priority setting and decision making within each functional group, from research to service, including monitoring and IM/IT. Furthermore, ongoing communication and interaction between all these functional groups should be maintained in order to ensure effective knowledge transfer.
The following recommendations are based on the findings and conclusions of the evaluation. The evaluation recommendations are directed to the Assistant Deputy Minister (ADM), MSC, in light of the ADM’s responsibility for the overall management of weather prediction activities.
Recommendation 1:
In order to support effective transfer of the MSC’s weather prediction science knowledge to end users, the following should be carried out:
In the context of available resources, a weather prediction client Service Strategy should be prepared that outlines a roadmap for the MSC’s future and ongoing weather prediction products and services.
The Strategy should be prepared in consultation with all functions involved in the weather prediction chain, including R&D, monitoring, production and IM/IT.
The Strategy should be communicated widely across all functions involved in the weather prediction chain.
A process should be put in place to ensure that the Strategy is reviewed and revised on a regular basis.
The evaluation found that the absence of a clear articulation of priorities for client products and services hinders effective priority setting and decision making along the weather prediction innovation chain. Also, there is a need to strengthen the interaction between the science and service functions in order to ensure that both the external client perspective and the possibilities offered by science are commonly understood and appropriately considered in priority setting.
Recommendation 2:
Identify clear roles and responsibilities to support the transfer, into operations, of knowledge developed in the national research laboratories.
The establishment of five national research laboratories was an important aspect of the MSC Transition initiative introduced in 2003. The scientific research of the laboratories was intended to benefit not only forecasters across the MSC, but also research organizations within and outside of Canada. However, evidence from this evaluation has identified a gap in terms of ensuring that research that is developed in these laboratories can be used by stakeholders. The gap points to lack of clarity on how, and by whom, this knowledge transfer function would be performed.
Recommendation 1
The ADM-MSC agrees with the recommendation.
Management Action
a & b) The MSC has undertaken the development of a documented Service Strategy. A draft was completed in early November 2010 for internal consultation with various functions components of the WES program (R&D, monitoring, prediction and IM/IT). The Service Strategy articulates goals and strategies along program activity priorities, which in turn will provide direction for science, monitoring and prediction development.
c) Engagement with staff on the Service Strategy began in November 2010 during a forum of service managers and staff, and will continue in a variety of forums and venues until the end of FY-10/11. Communication of the Service Strategy across all functions will be undertaken once the document is completed and approved by the WES Board (March 2011).
d) The MSC has created program committees organized along business lines (public, marine, aviation, etc). The program committees will articulate MSC priorities for the various client groups following the overall roadmap of the Service Strategy. As part of the annual program-planning cycle, the committees will make recommendations to the DG of Weather and Environmental Prediction and Services (WEPS) for revisions and updates to the Service Strategy and any such revisions will be presented to the WES Board for adoption. Program committees have membership from all the functional areas and important enabling functions such as CIOB. Additionally, the WES DG Leads Committee will continue to have the responsibility for providing oversight and ensuring horizontality of all WES Board governance bodies.
Timeline
Deliverable(s)
Responsible Party
Nov. 2010
March 2011
Ongoing
Service Strategy
Staff engagement
Regular PWEMP committee meetings
DG, WEPS, MSC
DG, WEPS, MSC
DGs, MSC
Recommendation 2
The ADM-MSC agrees with the recommendation.
Management Action
When the national laboratories were initially established, specific actions were taken in an attempt to facilitate the effective transfer of science outputs into our operational (prediction) environment. These actions included:
a) Co-locating the national laboratories with the EC storm prediction centres and with other EC operational centres such as the Canadian Meteorological Centre (CMC) and the Canadian Meteorological Aviation Centre (CMAC). This co-location provides an opportunity for R&D to connect to our external service interfaces through the service delivery mechanisms at the operational centres.
b) Establishing a cadre of managers working in the storm prediction centres to oversee training and science transfer activities, as well as to manage the involvement of operational meteorologists in research, development and implementation activities within the national labs.
c) Appointing a national manager of science transfer and training (STT), to ensure high standards of science validation and efficient and sustainable implementation practices, and to nationally coordinate activities,
d) Creating a Technology Transfer Advisory Committee (TTAC) to ensure that all aspects of science transfer projects, including well-defined and well-validated user requirements, are respected throughout the science transfer process.
Unfortunately, it appears that these actions were not sufficient to support an effective transfer of science outputs from the national laboratories into operations, owing to limited resources in key areas and the need to clarify governance. As such, additional changes were made more recently which we believe will build on this infrastructure and, once fully implemented and with sufficient time to become part of our ongoing processes, will demonstrate the desired results. These changes include:
e) To further focus MSC resources on specific and high priority client needs, a series of signature projects was tabled, with each project assigned to a senior MSC manager for execution.
f) To support the integrated delivery of weather services from its interdependent functions, including weather research, the role of the WES DG Lead Committee was clarified and enhanced. The Committee comprises membership of all of the integrated WES functions and serves as a forum for priority setting and decision making. As such, all WES DG leads share responsibility around the WES results. The WES DG Lead Committee will address all over-arching and horizontal elements of the WES Programs, including the transfer of R&D into operations.
g) In addition to the WES DG Lead Committee, WES has formed a Public Weather and Environmental Management Program (PWEMP) committee to ensure cohesion in national program delivery for the Weather Observations, Forecasts and Warnings Program (Program Activity Architecture sub-sub activity 2.1.1). It is an appropriate forum at which to discuss all aspects of the science-into-operations processes in support of Program 2.1 results.
h) To further strengthen the link between R&D and operations, the Meteorological and Environmental Predictions Innovation Committee (MEPIC) was created, co-chaired by the DG of ASTD and the DG of Weather and Environmental Operations (WEO). Formal membership also includes senior representatives of the internal client committee, as well as the DG of WEPS (who represents the interests of the external user community).
We believe that these initiatives, when combined with existing infrastructure, will ensure users see the benefits of science investments in their daily operations. Monitoring the status of the signature projects and WES deliverables, as documented in meeting minutes from these key committees, will provide evidence of the degree to which this is achieved.
Timeline
Deliverable(s)
Responsible Party
July 2011
Progress against deliverables identified in Signature project charters
Signature project managers
March 2011
MEPIC forward agenda
DG, ASTD, S&T
December 2011
Delivery on priorities
DG, WEPS, MSC
March 2011
PWEMP committee forward agenda
DG, WEO, MSC
December 2011
Delivery on priorities
1 Meteorological Service of Canada, Annual Report 2006.
2 Ian D. Graham, Jo Logan, Margaret B. Harrison, Sharon E. Straus, Jacqueline Tetroe, Wenda Caswell and Nicole Robinson, 2006, "Lost in Knowledge Translation: Time for a Map?" The Journal of Continuing Education in the Health Professions 26: 13-24, p. 17.
3 An EC Priority Management Board, comprised of senior managers, with the mandate to provide Canadians with world class meteorological and environmental information, predictions and services to ensure safety and to support economic activity.