Previous page | Table of Contents | Next page
The findings of this evaluation are presented below, by evaluation issue (relevance and performance) and the related evaluation questions. The findings at the overall‑issue level are presented first, followed by the findings for each evaluation question.
A rating is also provided for each evaluation question based on a judgement of the evaluation findings. The rating symbols and their significance are outlined below in Table 3. A summary of ratings for the evaluation issues and questions is presented in Annex 3.
Table 3: Rating Symbols and Significance
Symbol
Significance
Achieved
The intended outcomes or goals have been achieved or met
Progress Made; Attention Needed
Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed
Little Progress; Priority for Attention
Little progress has been made to meet the intended outcomes or goals, and attention is needed on a priority basis
N/A
A rating is not applicable
~
Outcomes achievement ratings are based solely on subjective evidence
Evaluation Issue: Relevance
Overall Findings: The provision of meteorological services is legislated within the Department of the Environment Act. R&D, production, monitoring and service delivery activities in support of weather prediction are aligned with federal government priorities. Furthermore, by supporting the provision of quality meteorological services and information, these activities also support the departmental mandate of EC and several federal stakeholders, and address the need to inform Canadians about their changing environment to help ensure their safety and welfare.
Evaluation Issue: Relevance: Q1. Continued need for activities
Indicator(s):
Methods:
Rating: Achieved
Multiple sources indicate that there is a continued need for R&D, production, monitoring and service delivery in support of weather prediction, because these activities allow for the provision of accurate weather prediction information and services, helping Canadians reduce risks posed by changing weather conditions. Moreover, there is a large and growing demand by Canadians for weather data.
According to the most recent National WES Products and Services Survey (2007), weather is the top subject of interest for Canadians who access the news. Nine in ten Canadians reported that they access weather forecasts at least once per day. A relatively large proportion of surveyed Canadians also reported that they needed the following types of weather information to make decisions or plans: general information about temperature (25%) and precipitation type (24%), information on road or highway conditions (17%), temperature highs and lows (16%) and precipitation in general (16%).21
The World Meteorological Organization (WMO) Convention, of which EC is a signatory, recognizes "the importance of an integrated international system for the observation, collection, processing and dissemination of meteorological, hydrological and related data and products." Furthermore, the Convention reaffirms "the vital importance of the mission of the National Meteorological, Hydrometeorological and Hydrological Services in observing and understanding weather and climate and in providing meteorological, hydrological and related services in support of relevant national needs which should include the following areas: (a) Protection of life and property, (b) Safeguarding the environment, (c) Contributing to sustainable development, (d) Promoting long-term observation and collection of meteorological, hydrological and climatological data, including related environmental data, (e) Promotion of endogenous capacity-building, (f) Meeting international commitments, (g) Contributing to international cooperation."22
As noted in the 2008 severe weather audit by the CESD, weather warnings can help minimize damages from costly severe weather events.23
Evaluation Issue: Relevance: Q2. Alignment of activities with federal government priorities and departmental strategic outcomes
Indicator(s):
Methods:
Rating: Achieved
Evidence shows the importance of weather prediction and its supporting science activities to federal government priorities and departmental strategic outcomes.
Although not a primary target of federal government budget investments over the past three years, weather prediction science is nevertheless a unique and critical support component of the government-wide priorities to ensure the safety and security of Canadians and contribute to their economic prosperity, as reiterated in the 2007, 2008, 2009 and 2010 speeches from the Throne. For example, weather prediction science innovations enable more accurate severe weather warnings, which in turn are essential to enable Canadians to protect themselves from the potential negative impacts of severe weather events. Weather prediction science was also a key contributor to the safe operation of the Vancouver 2010 Olympics. Furthermore, targeted weather prediction products and services support the efficient and safe operation of national defence, aviation, marine transportation, and other economic sectors.
Budget 2010 provided $9.2 million over two years to EC to deliver meteorological services in the Arctic in order to help meet Canada’s commitments to the International Maritime Organization.24
Documents also indicate that scientific activities in support of weather prediction are directly aligned with one of five strategic outcomes in Environment Canada’s 2010-11 PAA: "Canadians are equipped to make informed decisions on changing weather, water and climate conditions."25
Weather prediction activities also align with one of three strategic outcomes in EC’s Sustainable Development Strategy 2007-2009: "Weather and Environmental Services: Weather and environmental predictions and services reduce risks and contribute to the well-being of Canadians."26
Evaluation Issue: Relevance: Q3. Legitimate and necessary role for government
Indicator(s):
Methods:
Rating: Achieved
Documents underline the legitimacy and necessity of the government’s role in undertaking scientific activity in support of weather prediction.
There is a clear, legislated mandate for federal government involvement in the provision of meteorological services, set by articles 4 and 5 of the Department of the Environment Act. Weather prediction also serves other pieces of legislation, including the Canadian Environmental Protection Act, the Emergency Management Act and the Aeronautics Act.
Weather prediction science activities are necessary to meet Canada’s international obligations. For example, as mentioned above, Budget 2010 allocated funding to help meet commitments to the International Maritime Organization. Also, as a member of the WMO, Canada must provide resources and data that meet international standards in exchange for access to global data and information.27
Because of the global nature of weather phenomena, it is important that the federal government liaise with the global meteorological community. Information sharing and collaboration allow Canada to "leverage international efforts for maximum benefit to the Government of Canada and to Canadians."28
Scientific activity contributing to weather prediction also plays an important role in meeting domestic responsibilities. Weather services at EC contribute to fulfilling EC’s departmental mandate and to supporting the mandates of several federal stakeholders, such as Transport Canada, Public Safety Canada, and the Canadian Coast Guard.29
Evaluation Issue: Performance - Achievement of Expected Outcomes
Overall Findings: The evaluation found evidence that the needs of end-clients of weather prediction products and services are being adequately addressed by weather prediction science decisions, as shown by positive results of the most recent public opinion survey, the satisfaction expressed by interviewed clients, and the high regard in which Canadian weather prediction science is held by external collaborators. Recent changes made to governance mechanisms are expected to address some of the challenges encountered with integration of S&TB and CIOB in weather prediction science priority setting and decision making. Similarly, although still in early stages of implementation, the QMS is already benefitting weather prediction science decision making by requiring a more systematic documentation of processes and mapping of internal/client relationships, thereby facilitating science knowledge transfer across the innovation chain. The main gaps and limitations found pertain to the transfer of science knowledge within EC, particularly in terms of communication between the service and other functions along the innovation chain, and the transfer of science from the MSC’s national laboratories into operations. Additionally, more could be done in terms of enhancing the weather prediction products delivered and the delivery mechanisms to increase access to products for external clients.
Evaluation Issue: Achievement of Expected Outcomes :Q4. Internal and external client needs being met
Indicator(s):
Methods:
Rating: Progress Made; Attention Needed
In order to deliver improved services to external clients, those involved in the delivery of weather prediction need to be part of an integrated innovation chain, supported by the appropriate infrastructure and processes and collaboration from the global meteorological community. The needs of external clients of weather prediction are diverse, expanding and becoming increasingly sophisticated as a result of increasing possibilities due to evolving science and technology. Evidence shows that EC is well regarded among the international meteorological community, and is recognized as a valued contributor to scientific collaboration and a world leader in a number of key areas. Evaluation findings show that although traditional weather services are working well and continue to improve, more could be done to ensure that enhanced capabilities and data are shared with external clients in a manner that better supports decision making.
Weather prediction clients can be categorized as either internal (within EC) or external (end users of weather prediction products and services outside of EC). This section outlines the expectations and needs of each category of client, followed by a description of the circumstances in which weather prediction services are meeting these needs, and circumstances in which they are not.
Expectations and Needs of Weather Prediction Clients
The delivery of weather prediction services includes an integrated supplier/client relationship between the research, development, production, monitoring and services functions, supported by enabling functions-including IT provided via CIOB.
Interviewees were asked about the information, products and/or services they need in order to perform their role as part of the weather prediction innovation chain. Interviewees across all functional groups within that chain identified a relatively consistent list of needs, including:
access to IM/IT resources (via CIOB);
supercomputing capacity;
access to monitoring data, from an increasingly complex and diverse network of sources-domestic and international, public and private, and space-based and land-based;
collaboration with other weather professionals, including meteorologists, scientists and academics, both domestically and internationally;
a clear understanding of client needs;
a well‑managed technology transfer process;
long‑term vision and leadership;
an understanding of current priorities and deliverables; and
enabling functions, including human resources (HR), finance and procurement.
Representatives of the service delivery function also identified a need to be aware of emerging science or technological developments that can lead to new products or services in the future to support evolving client needs.
External Clients
The document review and interview findings revealed that as advancements in science and technology allow greater sophistication of weather prediction activities, external clients demand more detailed information in terms of coverage, depth and accuracy.30 This includes an expectation of continual improvements to forecasts and warnings of severe events, such as storms, so that people can be adequately warned and their safety can be assured.31 This finding extends to public and private sector users, who are both looking for increasing refinement and precision for their unique requirements, in order to optimize health, safety, economic and environmental benefits (e.g., providing more frequent readings on a variety of measures, including temperature, wind speed and precipitation, to emergency organizations responsible for responding to floods, fire situations or winter storms).32 As noted in a U.S. report that looks at the roles of various sectors in the delivery of weather and climate services, it is essential that the public be able to trust government-generated weather information, noting that "[t]rust results from five primary characteristics of the information: (1) accuracy, (2) reliability, (3) objectivity, (4) open and unrestricted access, and (5) scientifically based error estimates."33
Approximately half of the interviewees most familiar with external client needs (i.e., individuals in production and service roles, and external clients) reported that appropriate dissemination of weather prediction information is extremely important to external users, with a need to make the wealth of available data accessible. This includes providing access in convenient, user‑friendly ways, which may include use of web capabilities, graphics, and providing services for handheld devices. A growing demand for data overlays, (e.g., geographic information systems, Google Earth, web-mapping products) was also noted by interviewees, and in particular it was noted that emergency management organizations would like to see geo-referenced maps of a situation or hazard. This was supported by several key documents, such as a 2008 MSC consultation document,34 EC’s 2010-2015 Nowcasting R&D Strategy35 and an external benchmarking study on meteorological services business models conducted in 2009.36
According to external clients and service delivery representatives interviewed, many external clients have sophisticated needs. More than simply knowing the weather, clients wish to understand the impacts of weather. Clients are looking for "decision aids" to know, for instance, whether it is safe to fly a helicopter, whether a hangar door will open, what is the probability of the volume of rain reaching levels that will result in increased insurance claims, or whether conditions are ideal for spraying fertilizer. For the most part, this role is not provided by EC. Clients must interpret the data themselves, or obtain this capability from the private sector. The MSC has forwarded many requests for such services to the private sector, through the referral service of the Canadian Meteorological and Oceanographic Society (CMOS).37 Examples of these private sector requirements include construction companies needing meteorological information to understand the impact of weather on concrete, ski hill operators looking for information to better anticipate snow conditions, and the tourism sector and film industry sector looking for weather information to help plan their activities. EC’s website includes a link to the CMOS website, which includes a list of over 50 private sector organizations that can provide tailored solutions.38
Circumstances in which Weather Prediction Services are Meeting Client Needs
Interviews with domestic and international peers (e.g., academic scientists and representatives from other federal government departments or other countries’ meteorological services) reveal that EC’s MSC is well regarded and respected, both as an organization and for the many employees who are experts in their field. EC belongs to an international organization of top‑tier national meteorological services that collaborate and share scientific learnings. An independent review of the MSC’s R&D program notes that in some cases the MSC’s program components are the world leaders. Specifically, the review notes that the MSC’s capability in cloud physics instrumentation and data analysis is better than any other in the world, that its Global Environmental Model for numerical weather prediction represents the leading edge of the science, and that its Research Data Management and Quality Control System has received worldwide attention and been adopted by other agencies including the WMO, the World Data Centre for Precipitation Chemistry and the United States Environmental Protection Agency.39 Interviews within EC also reveal a high degree of mutual respect among weather prediction team members, with several interviewees recognizing their colleagues’ knowledge and commitment.
From both the internal and external client perspective, traditional weather prediction delivery generally works well and appears to be improving with the enhancement of atmospheric models. Research conducted by S&TB has resulted in the development of world-class numerical weather prediction and data assimilation systems. One result of the application of these systems has been an increase in predictive capability: the MSC is now able to make 7-day lead forecasts that are of greater quality than the 36-hour lead forecasts that were being produced in 1958.40 EC scientists supporting the MSC’s weather predictions have received awards at both the national and international level.41 Interviews with external clients revealed high levels of satisfaction with the quality of basic services provided and the responsiveness of the MSC to meet their needs. One of these clients stated that in terms of the quality of forecasts, "Canada is world-class." Results from a 2007 survey conducted with a random sample of Canadians revealed that Canadians are largely satisfied with the quality of weather information provided to them, with eight in ten Canadians saying they are satisfied with the accuracy of EC’s weather information and services, both in general and in terms of weather warnings.42
A minority of interviewees responsible for delivering weather prediction made note of EC’s high degree of automation, with a few interviewees reporting that the MSC is a world leader in this area. The CESD’s December 2008 Report noted that "Canada has been a world leader in producing highly automated regular forecasts."43 This is consistent with the MSC’s 2008 consultation document, which states that the MSC is "one of the most automated weather services in the world,"44 and with data reported in EC’s 2006-2007 DPR, where it was noted that Canada is "a world leader in using automated observing equipment."45
The following are other examples provided by interviewees of where weather prediction services are addressing client needs:
EC’s use of nowcasting during the 2010 Winter Olympics received positive feedback and excellent results from the client perspective. New technology was used to enhance the existing national weather prediction system.
An emergency forest fire situation in 2009 in British Columbia, where the MSC worked with provincial emergency measures personnel, was provided as an example where the full system worked well, including having the appropriate data, the forecasts and the services.
Implementation of the Radarsat technology to estimate wind speeds over oceans, which was identified as a good example of strong ongoing communication between science and operations to ensure training is in place in order to manage the technology transfer to forecasters.
Increasing success in identifying severe weather, although there are ongoing concerns with the timeliness of notification and the dissemination of information (see below).
Circumstances in which Weather Prediction Services are not Meeting Client Needs
Both the interview findings and the document review identified a continuing need to improve the dissemination of weather prediction data. A few interviewees noted that more data and scientific capability are available than are currently shared; the main challenge is to make this available to users in an accessible way in order to support their decision making. As one interviewee in a service delivery role stated, "from my perspective, CMC [Canadian Meteorological Centre], the prediction centre, [and] the science community are doing wonderful things. What we don’t see . is the organization getting this wonderful information out to the decision makers. This is the real struggle." This appears to be a well‑recognized challenge, because "Improving the usefulness and enhancing delivery of services and information for users and decision makers" was identified as a WES Board priority for 2010-11.46 Additionally, the development of a dissemination strategy was identified as a priority for the services team in 2009-10, although it was also noted that financial challenges could have serious impacts on the organization’s ability to make progress delivering on dissemination services.47 The need for effective dissemination was also mentioned in the CESD’s December 2008 report in its chapter on Managing Severe Weather Warnings.48 A 2010 follow-up of management actions in response to the CESD report indicated that plans have been initiated to deliver improvements in the distribution system for warnings as well as to gather information from end users on warnings’ effectiveness.
Two aspects of the dissemination challenge were consistently raised by interviewees: 1) making more data available to meet the increasingly complex needs of users; and 2) presenting data in a more user-friendly manner.
Data availability. As the complexity of user needs evolves, the range, type and frequency of data should also change to meet client needs. Examples cited by interviewees include a greater number of probabilistic products for private sector clients; a broader range of website icons for relaying probability forecasts to advanced users; and more frequent readings for emergency fire prevention clients.
Data user-friendliness. Interviewees spoke of various opportunities to improve the user-friendliness of data. This includes capitalizing on web capabilities (e.g., using more graphics over text), updating technology to display data in a more user-friendly format, and exploring additional delivery mechanisms (e.g., services to handheld devices).
A need for better access to support and training was also identified by a few interviewees, particularly if clients are responsible for their own value-added decision tools. As noted previously, specialized support for many private requirements is now being redirected to the private sector, with EC’s role shifting to providing access to data in standardized formats. One external client noted that much of the required expertise lies in EC, but there are no formalized training environments to share this knowledge so that clients are better able to leverage the information. Another interviewee (in a service delivery role) provided examples of two emergency situations where the provided forecasts were good, but the decision makers did not fully understand the implications and as a result the appropriate preventative measures were not taken. As noted previously, EC’s website provides links to a CMOS site that lists private sector providers for many of these services.49
This finding regarding access to support and training is also related to the MSC’s need to find the appropriate balance between its "public good" mandate and a growing demand for more commercial products and services, which is an ongoing challenge facing a number of international meteorological services.50 The MSC’s consultative document identifies questions regarding where the Department should draw the line in terms of providing data versus providing value-added products and indices, as well as how best to ensure that services are aligned with client needs. A few interviewees raised this issue under the context of needing a clear understanding of the service strategy.
Evaluation Issue: Achievement of Expected Outcomes: Q5. Science knowledge being transferred.
Indicator(s):
Methods:
Rating: Progress Made; Attention Needed
Departmental stakeholders involved in the delivery of weather predictions clearly identified with being part of an innovation chain, and recognized the benefits of collaboration both with internal colleagues and with the broader science community domestically and internationally. Internally, the interaction between research, development, production and monitoring functions appears strong, but more could be done to strengthen communications with the service function in order to ensure that the external client perspective is adequately considered in science decisions and to provide the service function with an improved understanding of emerging science in order to help anticipate what may be possible in the future. A gap was identified in the area of transferring science developed in the MSC’s national laboratories into operations.
In the context of this evaluation, knowledge transfer is defined as the complex process of obtaining scientific research used by stakeholders, or the process of moving research into practice. Literature on the subject reveals that there is little consistency in terminology used to describe this process, but does identify important concepts for consideration. In particular, the interactions and transfer between all stakeholders is critical, and should be collaborative and two-way; and furthermore, knowledge transfer extends beyond the first step of disseminating knowledge or information, to putting the knowledge into action.51 Table 7 provides the list of indicators used to evaluate knowledge transfer success,52 with ratings assigned based on evidence gathered during this evaluation.
Table 7: Knowledge Transfer Success Indicators and Ratings
| Achieved | Progress Made; Attention Needed | Little Progress; Priority for Attention | |
|---|---|---|---|
| Interactions between Stakeholders, including Engagement of the Target Audience | |||
| Communication channels, processes and context between knowledge translation actors | √ | ||
| Working relationships among stakeholders | √ | ||
| An ongoing forum for sharing among stakeholders | √ | ||
| Opportunities for collaboration | √ | ||
| Shared vocabulary among stakeholders | √ | ||
| Knowledge being relevant to and understood by the target audience | √ | ||
| Knowledge Use or Application | |||
| Knowledge being used to inform decision making | √ |
Internal Knowledge Transfer
Both at the working level and at senior management levels, the need for collaboration and good communication across the innovation chain is well recognized by interviewees.
The interaction between research, development and production appears strong, and S&TB scientists interviewed for the evaluation appear to feel like part of the weather prediction team, despite their organizational separation. There were, however, a few individuals who identified a need for stronger links between research (which organizationally resides in S&TB) and development (which resides in the MSC). Additionally, a few interviewees stated that monitoring could benefit from greater input from science, such as on how best to maximize the benefit from the monitoring network. The Meteorological and Environmental Predictions Innovation Committee (MEPIC), which was recently created, is intended to help strengthen collaboration between these roles, among other objectives.
The desire was expressed, from interviewees representing science (R&D) and service, for improved communication between the two groups to better understand and anticipate client needs. As one client interviewee noted, "that’s something we’re missing in the weather services--connecting the user with the bright ideas in the back room." The perspective from science was articulated by an interviewee who stated "We need to reach out to the potential users and partners to show them the possibilities. This process needs to be improved." Several interviewees expressed a desire for greater clarity on the service strategy, and a minority of interviewees in research, development and production roles felt that the link with services needed to be strengthened. "Improved interface with science" was also identified as a priority for the service team for 2009-10.53
Although a critical player in the delivery of weather predictions, CIOB was generally not viewed by key informants as part of the innovation chain, but rather as a key enabler. Within the context of discussing CIOB governance processes, a few interviewees expressed concerns with the CIOB model of client/supplier relationship for weather predictions, because IM/IT is so integral to the MSC. A few interviewees stated that longstanding personal relationships help the process work, but expressed concerns that working within this model might prove even more challenging in the future when these relationships no longer exist. One interviewee expressed the opinion that it was important for CIOB to feel like part of the weather service, including "part of the decision-making family." These comments would appear to be consistent with CIOB’s vision for its role within the Department as expressed in its strategic plan, where it states a desire to move "from a paradigm of delivering services to client specifications to more of a partnership, with a more intimate relationship between IM/IT and program areas."54
Departmental stakeholders involved in the delivery of weather prediction, interviewed as part of this evaluation, clearly identified with the concept of knowledge transfer and the notion of serving both internal and external clients. There was an identifiable use of common language among interviewees involved in the delivery of weather prediction, and a clear recognition that they are part of an innovation chain aimed at improving service to end users. The use of shared language among stakeholders and the importance of interaction have been identified in the literature as indicators contributing to success in knowledge transfer, because of their contributing role to support enhanced communication and information sharing among various stakeholders.55 One interviewee specifically credited QMS with the "paradigm shift" toward being a client-oriented service organization.
There is a knowledge/technical transfer gap associated with the national laboratories’ inability to transfer their science into operations. A few interviewees representing different roles in the innovation chain noted this shortcoming during the evaluation, identifying a need to shift the focus from development toward operationalizing the science transfer. This gap is also recognized by the Meteorological Research Division (MRD), which prepared a document outlining the computing requirements needed in the national laboratories to bring scientific developments into an operational forecasting environment.56 Interviewees had differing perceptions as to why this gap existed, including insufficient oversight in providing for this role at the time the laboratories were created, the science transfer and training (STT) role not occurring as envisioned due to competing priorities, and shortages in CIOB resources.
External Knowledge Transfer
Approximately half of interviewees within the MSC and S&TB observed an increased level of collaboration with external stakeholders over time. Supporting examples provided by interviewees were diverse, but included the existence of more formalized agreements, including Memoranda of Understanding with provinces and other government departments, cooperation with other international meteorological services, research with universities, and a free and open exchange of data (inter-jurisdictional), supported by a number of coordinating bodies (e.g., WMO, World Data Centre for Precipitation Chemistry), and an increased role for private sector organizations in the delivery of specialized products. A detailed analysis of the role of each of these players in the delivery of science in support of weather prediction was beyond the scope of this study.
International collaboration is an important part of knowledge transfer in meteorology, and the Department’s weather service benefits from collaboration on global science and shared access to global data. Under the coordination of the WMO, weather data collected by various countries’ national meteorological services are shared freely among members of the WMO,57 and numerous committees and organizations exist to leverage research globally. An interviewee from an international meteorological service praised EC’s role in the area of international collaboration, noting "They are a really strong player within our world community, in climate and weather research in particular. And if we’re looking for partners, inevitably, Canada is one of the partners we look to collaborate on a number of areas. I can’t speak more highly about their willingness to be good players..."
There is strong evidence to support that weather prediction is being used to inform decision making. A 2007 survey found that nine in ten Canadians make a point of looking at or listening to weather forecasts at least once per day. Furthermore, approximately one third of Canadian workers say their work always (21%) or usually (11%) requires them to make decisions based on the weather.58 And furthermore, nearly three in ten (28%) say that forecasts "always" provide enough information to make decisions or plans, while 52% say this is usually the case.59 Interviewees also provided numerous examples where weather prediction played an important role in decision making. The use of weather prediction to schedule events during the 2010 Olympics, to plan for the control of forest fires, and to assess opportunities for wind energy were among examples noted.
Evaluation Issue: Achievement of Expected Outcomes: Q6. Effectiveness of priority setting and decision making
Indicator(s):
Methods:
Rating: Progress Made; Attention Needed
Several mechanisms are in place for weather prediction priority setting and decision making. These mechanisms appear to generally function well, but there is a perception, particularly among working level managers and staff, of a lack of long-term coordinated vision and priority setting for activities involving weather prediction science research and IM/IT support functions. As is often the case for cross-cutting functions, engagement of S&TB and CIOB in decision making and priority setting has been challenging given their place in the organization (part of WES Board, but not the MSC). The creation of MEPIC is seen by several stakeholders as a welcome solution to the absence of a hierarchical decision‑making structure for science activities delivered across both branches; and, combined with the ADM‑MSC’s expected launch of new signature projects is expected to address the need for long-term coordinated vision and priority setting.
This section first documents the mechanisms that are currently in place for priority setting and decision making on weather prediction science activities, then identifies the sources of information and criteria used as part of these mechanisms for priority setting and decision making, and, finally, examines the adequacy and effectiveness of these mechanisms.
Existing Priority‑Setting and Decision‑Making Mechanisms60
High‑level priorities for weather prediction science (as well as other decisions pertaining to the MSC’s activities) are set by the WES Board, which is chaired by a Board Secretary and composed of 11 ADMs and DGs, including the ADMs for the MSC, S&TB and CIOB. The WES Board is responsible for:
Ensuring that the Department is making progress toward the Strategic Outcome "Canadians are equipped to make informed decisions on changing weather, water and climate conditions" and is delivering on program results.
Providing direction, priority setting, planning and reporting, as well as recommending resource allocations for the Strategic Outcome and related program activities.
Ensuring that there is horizontal coordination on policy and program issues, addressing issues, and referring them to the Executive Management Committee as appropriate.
The WES DG Leads Committee is the main DG-level committee responsible for supporting the WES Strategic Outcome. Chaired by the DG of MSC’s Business Policy Directorate, it includes all MSC DGs as well as DGs responsible for ASTD, the Major Projects and Supercomputing Directorate (in CIOB), the Infrastructure Operations Directorate (CIOB), and Assets, Contracting and Environmental Management Directorate (Finance and Corporate Branch), as well as the WES Manager for Financial Services, the CIOB Portfolio Manager for WES, and the WES Board Secretary. This committee is the main venue under the current EC PAA for policy, program and planning discussions affecting the entire WES Program, including science-related activities. Committee members discuss and attempt to reach consensus on priorities before these are recommended to the WES Board. The DG Leads Committee also meets on a quarterly basis with the CIOB DGs to discuss joint WES/CIOB priorities and plans as well as issues of mutual concern.61
More precise priorities for the scientific component of weather predictions are set by MEPIC, co-chaired by the DG of WEO Directorate in the MSC and the DG of ASTD in S&TB. MEPIC also includes both branches’ directors responsible for research, development, operations and services as well as their respective section chiefs. The committee examines the entire "innovation chain", that is, the end-to-end process from science to operations to services. Its mandate is to look at future directions and resource gaps, make decisions on implementation of workplans and business processes, prioritize and decide on internal allocation of science‑related funds within the MSC or S&TB. MEPIC is also tasked with developing and presenting medium- and long-term innovation strategies to the WES Board. This committee was created in 2009 to replace the R&D/Canadian Meteorological Centre Committee. Key changes included the introduction of DG-level chairs and co-chairing of the committee by representatives from both the MSC and S&TB. Such changes were intended to strengthen the link between science and operations.
Priorities for numerical weather prediction (NWP) are set by the Comité des passes opérationnelles et parallèles (CPOP). This committee is chaired by the Director of National Prediction Development in the WEPS Directorate of the MSC, and is composed of a cross-section of directors, section heads, and senior scientists from the MSC, S&TB and CIOB. The committee is the main body managing the transfer, to operations, of R&D products related to numerical data predictions. CPOP examines concrete proposals and plans, approves changes to the operational system, considers and quantifies possible improvements to processes, and makes effort allocation decisions.
Priorities for non-numerical weatherprediction (non-NWP)62 are set by the Technology Transfer Advisory Committee (TTAC). TTAC "oversees the implementation of most emerging non-NWP science, and related technological development in MSC Prediction programs, including those that adjoin other MSC programs such as Service."63 This committee mirrors the CPOP but focuses on non-NWP for the regions. It does not have decision-making authority but, rather, recommends technical transfer priorities to the WEPS DG, who has final decision-making authority.
Information Sources and Criteria used for Priority Setting and Decision Making
The following sources are among those identified, through document review and interviews, as being used by the various priority-setting and decision-making committees to establish priorities and make decisions: annual Government priorities as outlined in speeches from the Throne and federal budgets, EC’s 2007 Science Plan, internal and external audits, program evaluations, research, reviews, comments and inquiries submitted to the National Inquiry Response Team, and input from weather prediction product users and stakeholders gathered through ad hoc consultations.
Representatives of the service function also acknowledged the need for priority setting in order to find the appropriate balance between "science push" and "client pull." In addition to ensuring that client needs are adequately considered in science decisions, as noted in section 4.2.1, representatives are also looking to understand which new innovations and capabilities are possible from emerging science, in an effort to anticipate how this may address the future needs of clients.
A few key informants observed that some science activities are pursued without being prompted by a request or need expressed by users. The two main reasons evoked by S&TB key informants to justify this were:
Given the time it takes to develop new applications and products, it is important to anticipate and address future needs in light of technological advancements. The move from monthly to seasonal forecasting is an example of a research initiative undertaken by core research in S&TB without being specifically prompted by the MSC. Such a decision was made, according to a departmental representative, because researchers had the scientific capacity to pursue this initiative and anticipated that one of the clients of weather predictions, the agriculture sector, would benefit from this innovation.
Some research should be pursued to remain at the cutting edge of scientific and technological advancements, in order to:
honour Canada’s responsibility to the international community;
maintain Canada’s position in the top‑tier countries in order to continue benefitting from the exchange of intelligence and expertise with other leading countries, and from free access to data sources that are an essential complement to and backup for Canadian monitoring data; and
offer a motivating and rewarding work environment in order to attract and retain competent scientists in a context of increasing international competition and an aging scientific workforce.
The above arguments were supported by several document sources:
In 2002, an independent panel review of the MSC R&D program highlighted the importance of international collaboration and communication. "Weather forecast and climate models, observational and data assimilation systems, and in fact the entire forecast production process are becoming more and more complex. At the same time, the resources in many countries for weather and climate research and operations are decreasing. These two factors mean that it is more important than ever for research and operational organizations to communicate and collaborate. More rapid progress of all national weather services can occur through the sharing of ideas, best practices and in some cases software.64" It also mentioned the risks posed by demographic gaps in EC’s science professional staff. "[...T]he professional staff at the MSC is aging as reductions in resources have slowed or eliminated hiring of younger people in the past decade. As a result, some of the programs and activities of the MSC have demographic gaps that can lead to serious shortfalls in scientific expertise, experience and leadership in the near future.65"
The importance of maintaining an active Canadian contribution to the international atmospheric research community was further reiterated in 2003 in the MSC’s first R&D strategic plan. "Atmospheric science issues are becoming increasingly globalized and more complex. They require so much data that they cannot be tackled by any individual country without stressing their research capacity to the limits. Canada will have to increase its contribution to the resolution of these issues, particularly in regards to data. This will enable Canada to keep its largely free access to unlimited sources of data and knowledge that are essential to solving its own national environmental challenges."66
The current global shortage of skilled meteorological researchers was echoed by authors of the external benchmarking study completed in 2009. "Met[eorological] Services everywhere are faced with the challenge of attracting and maintaining a skilled work force. Ongoing capacity building, as well as access to flexible and tailored HR policies and supports, will be critical to addressing this challenge, particularly as greater private sector opportunities emerge for these highly skilled workers.67"
The 2008 CESD audit of EC’s severe weather warnings endorsed the importance of Canada’s contribution to international meteorological scientific‑improvement efforts. "Over the past decade, advances in meteorology and information technology science have had a notable impact on the capacity and effectiveness of weather forecasting. Rapid advances in hardware and software technology have allowed Environment Canada and other weather services around the world using worldwide weather data to improve computer-modelling techniques to the point that they now form the basis for all weather forecasting.68"
Adequacy and Effectiveness of Existing Mechanisms
When asked to describe current priority‑setting mechanisms for their weather‑prediction‑related activities, key informants’ opinions were evenly split, across functional groups, as to whether these mechanisms are effectively functioning, not functioning, or partly functioning.
Some trends were observed when analyzing interview responses by hierarchical level:
Senior managers (directors and above) were generally more satisfied with the effectiveness of priority‑setting and decision‑making mechanisms. In particular, the creation of MEPIC was perceived as a welcome solution to the absence of a hierarchical decision‑making structure overseeing both S&TB and MSC weather prediction activities. Co-chairing of the committee by DGs from both the MSC and S&TB was seen as the best possible approach to provide oversight for multiple directorates/branches.
Working‑level managers and staff tended to be somewhat less satisfied than senior managers with priority‑setting and decision‑making mechanisms, perceiving that there is a lack of long-term coordinated vision and priority setting--or at least of a vision that is viewed as "realistic" within resources.
The main areas for improvements noted by key informants across categories were:
the need for better integration of the various functions (e.g., research, development, production, service, monitoring, IM/IT support) in decision making, especially IM/IT support and research, due to these functions’ location in separate organizational structures (CIOB and S&TB) outside of the MSC; and
the need for clearer communication of priorities and authoritative decision making on the part of senior management, especially regarding smaller and ad hoc project activities (larger projects benefitting from clearer decisions made at WES Board). When asked about barriers or issues that may need to be addressed in terms of meeting their weather‑prediction‑related needs and priorities, nearly half of all interview respondents mentioned improved governance and one quarter mentioned improved priority setting.
A key recurring theme across interview responses was the challenge of determining how best to distribute scarce CIOB IM/IT resources (i.e., supercomputer capacity and specialized technical support). Three key messages emerged from interviews:
CIOB representatives expect the MSC and ASTD decision makers to establish their own priorities regarding supercomputer use and specialized technical support, and to communicate these priorities to CIOB decision makers as well as MSC and S&TB managers and staff.
The MSC and S&TB expect CIOB to be accountable for the allocation of supercomputer and specialized technical support resources, and to provide feedback to managers and staff on which priorities were addressed and which ones were dropped, given limited resources.
Current decision-making processes are not fully effective in meeting either of these sets of expectations. This is attributed in part to:
the absence of departmental priorities regarding the distribution of CIOB resources across the Department’s various branches; and
the absence of a clear process or criteria to prioritize MSC requests for CIOB support, particularly for smaller, ad hoc projects, although a CIOB Portfolio Manager was recently assigned to the WES Board, which is expected to improve discussion of priorities versus capacity across the three organizations (i.e., MSC, S&TB and CIOB).
These observations are consistent with findings from the Audit of Governance of Specialized IT Resources, where the need for more clarity around the governance structure for the delivery of IT services was noted, along with the absence of mechanisms to allow competing priorities from different boards for the use of finite CIOB resources to be prioritized and resolved.69
The following recent/current improvements have been or are being implemented, and are expected to address the above-mentioned needed improvements:
The creation of MEPIC is expected to provide better integrated decision making across the S&TB and MSC functions along the innovation chain.
The inclusion of the CIOB ADM and CIOB Portfolio Manager on the WES Board is expected to improve decision making regarding allocation of IM/IT resources to weather prediction activities.
The implementation of new program boards under the WES Board is intended to fill a gap in governance arising from the disbanding of the Department’s former results management structure, by introducing a program (PAA element) perspective to complement existing organizational and functional perspectives. Each program board will have an accountable DG.
The ADM-MSC is currently overseeing development of action plans for a new vision and strategic directions for the MSC. Following a white paper consultation conducted with the MSC and its partners in 2008-09, which was communicated through a series of town hall sessions with MSC employees, managers and partners in the spring of 2010, the ADM-MSC has initiated the development of seven signature projects for "becoming the Meteorological Service of tomorrow." The seven projects will be as follows:
Developing a Monitoring Strategy - implementing a network of networks approach;
Implementing a High Performance Computing Strategy;
Enabling EC’s Science Plan (integrated monitoring and environmental prediction);
Planning for the Next Generation Weather Forecasting System;
Designing a Water Cycle Prediction Service;
Modernizing the Weather Warning and Service Delivery System; and
Developing a Climate Services Strategy.70
When asked for examples of good practices in setting priorities, a few key informants mentioned CPOP as being an effective committee; a few others mentioned that priority setting functioned well at the working level; while a few others mentioned MEPIC as a welcome addition.
When asked about their opportunities, as users of weather prediction products/services/information, to provide feedback and suggestions to decision makers in terms of their needs and priorities, a majority of interview respondents commented positively on such opportunities.
Evaluation Issue: Economy and Efficiency: Q7. Contribution of QMS to effective science knowledge transfer
Indicator(s):
Methods:
Rating: Progress Made; Attention Needed
Evidence indicates that the QMS has begun contributing to effective science knowledge transfer in support of weather prediction. Although implementation challenges have been encountered, some benefits are emerging, such as more systematic documentation of processes and mapping of internal/client relationships, thereby facilitating knowledge transfer across the innovation chain. There was an identifiable use of common language among interviewees involved in the delivery of weather prediction and, although it is not possible to say to what degree this is attributable to the QMS, there was a clear recognition by these interviewees that they are part of an innovation chain aimed at improving service to end users. In general, interviewees perceived that the QMS was most useful for areas of work with clear measurable and targetable objectives (e.g., operations), while implementation of the QMS was perceived to be more challenging in areas dealing with ideas or concepts that are difficult to quantify or measure (e.g., policy, high‑level strategic planning).
Quality Management System Implementation71
According to interviewees and document review findings, the MSC registered its projects to the ISO standard in response to WMO awareness‑raising efforts related to formalizing quality management systems maintained by its members and in anticipation of possible future certification requirements. The QMS was expected to help validate that MSC processes are rigorous and meet a recognized international standard, enhance the integrity of operations, contribute to improvements, and support MSC/WES decision making. Management oversight for QMS is provided by a QMS Steering Committee chaired by the ADM-MSC and composed of the responsible DGs.
Departmental documentation states that the ISO 9001:2000 standard is "currently the leading standard worldwide for national meteorological and hydrological service providers" and that "[t]his certification is important for the production and dissemination of departmental R&D under the WES Board, which includes programs related to atmospheric research as well as meteorological and hydrological monitoring."72
Although evidence suggests that QMS has generally been implemented as intended, a number of implementation challenges were noted during interviews and in documentation. Examples include:73
the time for implementation, which took longer than expected;
the level of effort and resources required;
a lack of strategic focus and a lack of full integration with the planning process;
some difficulties in obtaining buy-in from participating groups, particularly earlier in the implementation; and
QMS process maps initially being developed at a lower, more granular level, with development of an overarching process map being pursued afterwards, when the more detailed information proved to be too voluminous for decision-making purposes.
One key informant who was actively involved in the QMS development and implementation commented that the past three years were dedicated to launching and feeding the system with required information, with not much benefit from the system being felt by the programs, but that this year’s focus is for the system to start feeding information back.
Some of the delays encountered in implementation were due to the remapping of the QMS for a new PAA introduced in April 2010.74 This remapping is, however, expected to improve the user-friendliness of the QMS. According to documentation, it will be "[s]impler and easier to understand," will be "[b]ased on generic, overarching processes focussing on key functions," will show "stronger links between the policy, science, monitoring, prediction, and delivery functions" and support "better quality objectives and key performance indicators."75
Circumstances in Which QMS Works
When asked about the circumstances where QMS supports and empowers them to fulfill their role, approximately half of interviewees mentioned that the documentation of processes required by the QMS was a benefit, namely because they introduced order, rigour and/or clarity in the articulation of processes, roles and responsibilities.
Similar information was found in internal program documentation, which listed the following as emerging strengths:
"[The] [l]inked process approach of this standard has increased appreciation for the entire 'meteorological enterprise’; [it] provides a broader, more strategic picture to all staff of their results contributions";
"Improved employee moral [sic], pride and engagement - reinforced 'quality is top of mind’";
"Improved documentation and records (aids training and succession planning); consistent focus on training (planning and tracking)";
"Increased resolution of long-standing problems"; and
"Increased focus on services standards".76
Interviewees provided some examples of where it was believed that the QMS was particularly suitable for specific types of weather prediction activities. Examples included technology and dissemination areas, monitoring, operations or areas tied to a 24/7‑type of operation, and, more generally, areas where there are clear, measurable, targetable objectives.
A few interviews reported that the QMS had been helpful for researchers, namely because it introduced a better way of documenting what they do and of organizing their documentation.
Interviewees provided examples of factors perceived to have advanced the successful implementation of QMS, including the establishment of a QMS coordinator position, being well informed prior to implementation, ADM‑level commitment to ISO, and a sequenced/stepwise approach to implementation.
Circumstances in Which QMS Does Not Work Well
A number of areas were identified by interviewees where the QMS is less likely to function well. These include areas involving high‑level strategic planning, policy, services and, more generally, engagement, communication and areas dealing with ideas or concepts that are difficult to quantify or measure. Examples of explanations provided included:
the need to be reactive (for policy);
high‑level strategic planning and decision making are less process oriented;
challenges in identifying relevant indicators for policy, research or service; and
time (e.g., some projects take 8-10 years to complete, and, in the first years, information to measure and report may be limited).
An interviewee indicated that a peer review process exists for scientists and that QMS may not be appropriate for all aspects of science. Challenges in creating quantifiable, measurable, meaningful metrics were also noted during interviews.
Interviewees provided examples of factors perceived to have impeded the successful implementation of QMS, including a lack of resources for implementation, challenges in working with other parts of the Department that are not under this QMS (e.g., Human Resources, Finance), challenges in getting buy-in or support, and the time and/or effort required.
Link to Knowledge Transfer and the Innovation Chain
As noted previously in this report, departmental stakeholders clearly identified with the knowledge transfer concept and the notion of serving both internal and external clients. There was an identifiable use of common language among interviewees involved in the delivery of weather prediction and, although it is not possible to say to what degree this is attributable to QMS, there was a clear recognition by these interviewees that they are part of an innovation chain aimed at improving service to end users.
Evaluation Issue: Achievement of Expected Outcomes: Q8. External factors
Indicator(s):
Methods:
Rating: N/A
The stability and overall level of government funding is an important factor, external to the weather prediction innovation chain, which affects the decision making and activities of science in support of weather predictions. Availability of financial resources also plays a role in other external factors affecting weather prediction, which include challenges in obtaining IT resources and skilled human resources generally, as well as the need for a more robust supercomputing and monitoring infrastructure.
The capacity of R&D, monitoring, production and service teams to effectively support the development and delivery of weather prediction products and services was affected by several factors that are outside of the direct control of the respective EC branches involved. Described below, these factors should be taken into account when interpreting the evaluation findings.
Resource availability, and more specifically the stability and overall level of government funding,was identified as an external factor that has a significant influence on the capacity of research, development, monitoring, production and service to support weather prediction. This was raised as a key challenge by a majority of interviewees, and also identified in several documents, including the MSC’s 2008 consultation document, where the absence of needed funds was noted as challenging the Department’s ability to take advantage of advances in science and technology.77 It was reiterated in the 2009 external benchmarking study, which observed that the "MSC has continued to experience funding constraints and challenges and has not been able to keep pace with its capital infrastructure and human resource requirements."78 The benchmarking study also identified funding as one of the most significant ongoing challenges for virtually all of the international meteorological services examined.79
Challenges obtaining information technology resources were identified by most interviewees involved in the delivery of weather predictions in the MSC or S&TB. This issue was raised either as a general perceived shortage of IT resources, or as due to inefficient processes. Concerns with resource shortages were greatest among those working in research or development roles, where a shortage of IT development support was seen as limiting their ability to put science into practice. Documentation provided by some program representatives also identified this issue and provided a list of examples where the team had been affected by a shortage of CIOB resources or other enablers.
Human resources challenges were flagged by approximately half of interviewees within the Department involved in the delivery of weather prediction, with the main issue raised being a shortage of individuals with highly specialized skills, as well as challenges classifying and filling positions. The MSC has recognized this challenge and developed an HR plan to address the most critical gaps.80 Workforce renewal was also identified in the MSC’s vision document as a top consideration in planning for the organization’s future.81 As well, attracting and maintaining a skilled workforce was identified in the external benchmarking study as a challenge faced by meteorological services everywhere.82
Infrastructure challenges, particularly regarding a need for greater supercomputer capacity and for resources to address the "rust-out" of the monitoring network, were also identified by a few interviewees. The deterioration of the monitoring network was additionally identified in the CESD’s December 2008 audit report, as was the need for a long-term strategy for the MSC’s monitoring networks, supported by a capital plan.83 Interviewees frequently acknowledged that this issue is clearly recognized by senior management, and a 2010 follow-up of management actions in response to CESD recommendations identified that a plan has been prepared which outlines the steps to address gaps in EC’s monitoring capacity; however, interviews with senior management indicated that challenges remain in securing the required funds for such expenditures. This is a significant concern, because developments in the science that is needed to improve the products and services delivered to end users rely on increasingly complex models populated with ever‑expanding inputs from the monitoring side, and ultimately result in a need for greater computing capacity.84 Infrastructure concerns not only affect the Department’s ability to meet emerging demands for new products, but also put the provision of general forecasts at risk, as EC has moved to a highly automated system that is reliant on both a strong monitoring network and sufficient computing capacity. This is another area where the Department’s MSC is not alone, as the external benchmarking study identified that challenges associated with modernization and infrastructure renewal were faced by all meteorological services studied.85
EC relies on certain external organizations outside of the Department for monitoring information. Although this is a cost-effective approach, it does carry some risk. The MSC’s consultative document points out that becoming dependent on external groups for essential infrastructure or the delivery of critical services could become problematic if those agreements change in the future.86 The CESD’s December 2008 report also identifies the risks posed by EC’s current strategy of relying on external organizations as a significant challenge. In addition, a few key informants noted the drawbacks of relying on external organizations, such as lengthy and complicated contracting processes, and occasions when data are not delivered. Furthermore, external organizations may collect data in a manner that meets their own specific objectives, but does not necessarily address all of the Department’s needs (e.g., may not capture data at the level of precision most beneficial for the Department’s modelling).
Evaluation Issue: Performance - Demonstration of Efficiency and Economy
Overall Findings: Evidence suggests that the level of effort expended for weather prediction activities yields commensurate or better value for given resources. However, concerns regarding resources and sustainability were noted. Weather prediction science-related activities were generally perceived by a number of interviewees as being efficient; however, further collaboration with others (e.g., other organizations and/or other countries) and sharing of infrastructure have been highlighted as potential means of pursuing further efficiencies.
Evaluation Issue: Efficiency and Economy: Q9. Value for money and efficiency
Indicator(s):
Methods:
Rating:
A majority of interviewees consulted on the value for money of weather prediction science activities perceived that the level of effort expended yields commensurate or better value. Examples of views expressed included that weather prediction activities are competitive internationally despite having comparatively fewer resources than other groups, and that there is a large return for money spent. However, concern regarding resources and sustainability in the future was indicated.
An external benchmarking study that looked at meteorological service organizations in seven countries, including Canada’s MSC, contained an international cost comparison among the meteorological services studied, shown in Figure 4.87 The figure shows relative spending, not only in terms of the cost per capita, but also the cost per square kilometre.
Figure 4: Relative Spending on Meteorological Services88
Figure 4 indicates that Canada spends approximately twice as much per person as does the United States, although approximately half as much as Finland. Canada’s expenditures per square kilometre are indicated as the lowest among the countries surveyed. There is great variation within this category, however, with Australia and Canada each spending several times less per square kilometre on weather services than do the other countries surveyed.
It was noted in this benchmarking study that the MSC spending information reflects Canada’s relatively small population and relatively large land base. Although this information suggests good value for money, given the size of the territory being covered, definitive conclusions regarding value for money cannot be drawn solely from the information provided.89
A number of interviewee respondents generally perceive weather prediction activities as efficient. However, some suggestions for ways to improve efficiency were provided. Although half of the interviewees mentioned that collaboration had improved over time, a few interviewees suggested that the MSC pursue further collaboration with others (e.g., other organizations and/or other countries), through mechanisms such as a network of networks approach or further international leveraging or collaboration with other organizations. The possibility of looking at a different model or setup for the data centres was also suggested (e.g., looking at further consolidation of IT).
The external benchmarking study also noted a number of trends across world meteorological organizations, some of which may lead to greater efficiencies. For example, the study noted a trend in "growth in cooperative efforts with local agencies in order to achieve more integrated warning services, particularly for specific sectors... in turn leading to economic benefits in terms of avoided costs from these more specialized risk prediction services."90 In addition, an international trend toward some infrastructure sharing to maximize cost efficiency was noted.91 An international peer noted benefits of formal collaboration with other organizations internationally, because work such as modelling is becoming so complex.
Furthermore, the external benchmarking study indicates that many of the meteorological services they examined shared core challenges that the MSC faces (e.g., core funding reductions, increased cost-efficiency and accountability requirements, and increased demands for innovative delivery of new services). The study indicated that the meteorological services were working to determine how best to respond to these demands, and provided examples of efforts, including "maximizing integration of their core functions, establishing mechanisms to understand and respond to client and stakeholder needs, clearly demonstrating and communicating performance, and generating revenue from targeted services."92
21 EKOS Research Associates Inc., 2007 National WES Products and Services Survey Final Report, April 2007, p. 4.
22 Convention of the World Meteorological Organization, 2007 (1947), p. 9.
23 OAG, 2008 December Report: Chapter 2--Managing Severe Weather Warnings, p. 3.
24 Finance Canada, Budget 2010: Leading the Way on Jobs and Growth, March 4, 2010, p.106.
25 EC, Environment Canada’s Proposed 2010-11 Program Activity Architecture.
26 EC, Environment Canada’s Sustainable Development Strategy 2007-2009, 2006, p. 4.
27 MSC, "Forecasts for Canadians, Foresight for Canada - Articulating a Vision and Strategic Direction for Environment Canada’s Weather and Environmental Services," 2008,p. 7.
28 Ibid, p. 7.
29 Ibid, p. 20.
30 Ibid, p. 8-10.
31 "Stormy weather ahead: getting the word out," Regina Leader-Post, July 14, 2010, p. B9.
32 EC Weather and Environmental Services Board, 2010-11 Departmental Priorities, April 2010, (Presentation deck), p. 6.
33 National Academy of Sciences, "Fair Weather: Effective Partnerships in Weather and Climate Services," 2003, p. 25.
34 MSC, "Forecasts for Canadians, Foresight for Canada," 2008, p. 10.
35 EC, Nowcasting R&D Strategy, 2010-2015, undated, p. 13.
36 Allison Kerry et al., Benchmarking Study on Business Models for the MSC-Final Report, June 4, 2009, p. 5, 20.
37 EC, 2007-2008 Departmental Performance Report, 2008, p. 33.
38 EC, "Canadian Meteorological Private Sector," April 19, 2010. http://www.ec.gc.ca/meteo-weather/default.asp?lang=En&n=8DB56DCD-1
39 Independent Review of the Research and Development Program of the MSC, p.3.
40 Environment Canada’s Science Plan, vol. 2: Supporting Documents, 2007, p. 27.
41 EC, S&TB, Measuring Environment Canada’s Research and Development Performance, 2009, p. 17. http://www.ec.gc.ca/doc/scitech/mecrdp_e.html
42 EKOS Research Associates Inc., National WES Products and Services Survey 2007, Final Report, April 2007, p. 83.
43 OAG, 2008 December Report: Chapter 2--Managing Severe Weather Warnings, p. 6 (2.13).
44 MSC, "Forecasts for Canadians, Foresight for Canada," 2008, p. 6.
45 EC, 2006-2007 Departmental Performance Report, 2007, p. 44.
46 EC Weather and Environmental Services Board, Weather and Environmental Services, 2010-11 Departmental Priorities, April 2010, (Presentation deck), p. 11.
47 WES Board, "Delivery of Weather and Environmental Services (OPG 2B2), Priorities and Challenges for 2009-10," Presented by Diane Campbell, March 31, 2009, (presentation deck), p. 2, 6.
48 OAG, 2008 December Report: Chapter 2--Managing Severe Weather Warnings, p.20
49 EC, "Canadian Meteorological Private Sector," April 19, 2010. http://www.ec.gc.ca/meteo-weather/default.asp?lang=En&n=8DB56DCD-1
50 Kerry et al., Benchmarking Study, p. 22.
51 Graham et al., "Lost in Knowledge Translation,"
52 Indicators were derived from: Colleen M. Davison, "Knowledge Translation: Implications for Evaluation," in Judith M. Ottoson and Penelope Hawe, editors, Knowledge Utilization, Diffusion, Implementation, Transfer, and Translation: Implications for Evaluation - New Directions for Evaluation 124: 82-83.
53 WES Board, "Delivery of Weather and Environmental Services (OPG 2B2), Priorities and Challenges for 2009-10," March 31, 2009, p. 5.
54 EC IM&IT Plan, CIOB, "Simply the Best," Version 1.08, December 11, 2008, p. 10.
55 Davison, "Knowledge Translation: Implications for Evaluation," p. 79-82.
56 MSC, MRD, Computing Requirements for Regional Science Divisions and Associated Labs, November 2009.
57 OAG, 2008 December Report: Chapter 2--Managing Severe Weather Warnings.
58 EKOS Research Associates Inc., National WES Products and Services Survey 2007, Final Report, April 2007, p. 19.
59 Ibid., p. 83.
60 The ensuing description of existing priority‑setting and decision‑making mechanisms was developed using information from several sources, including key informant interviews as well as terms of reference and records of decisions for the various committees listed.
61 WES Board, Proposed 2010-11 PAA Governance S.O. 2.0 "Canadians are equipped to make informed decisions on changing weather, water and climate conditions," revised September 30, 2009.
62 NWPs are produced by inputting weather data into mathematical models, whereas non-NWPs are produced using other scientific means.
63 MSC, TTAC - Terms of Reference, 2010.
64 Friday, Jr. (Panel Chair) et al., Independent Review, p. 18.
65 Ibid.
66 MSC, Strategic Plan 2003-2012, Research and Development Program, p. iv.
67 Kerry et al., Benchmarking Study, p. 22.
68 OAG, 2008 December Report: Chapter 2--Managing Severe Weather Warnings, p. 8.
69 EC, Audit of Governance of Specialized IT Resources, 2010, p. 5-7.
70 EC Weather and Environmental Services Board, Weather and Environmental Services, 2010-11 Departmental Priorities, April 2010, (Presentation deck).
71 Sources include interview findings and the following two documents: MSC - QMS Office, WES Quality Management System (QMS), March 18, 2008 (Presentation deck), slides 3 and 4 [labelled as draft at bottom of slides]; Environment Canada. 2007-2008 Report on Plans and Priorities, p. 35.
72 EC, Measuring Environment Canada’s Research and Development Performance, 2009, p. 22.
73 Sources include interview findings and the following document: MSC, Performance Management, Joanne Volk, Weather and Environmental Services Board Quality Management System: Management Review (Presentation deck), slide 4, May 26, 2009.
74 WES Board, PAA, PMF, QMS, MRSS.Beyond the Alphabet Soup: Information Session for Employees and Managers of the Weather and Environmental Services (WES) Program (Presentation deck), slide 18 (labelled as draft at bottom of slides), March 2010.
75 Ibid., slide 21 .
76 MSC, Performance Management, Joanne Volk, Weather and Environmental Services Board Quality Management System: Management Review, (Presentation deck), slide 4.
77 MSC, "Forecasts for Canadians, Foresight for Canada," p. 11.
78 Kerry et al, Benchmarking Study, p. 6.
79 Ibid., p. 22.
80 MSC, The Meteorological Service of Canada’s People Plan 2009-2012, June 5, 2009.
81 MSC, "Forecasts for Canadians, Foresight for Canada," p. 11-12.
82 Kerry et al., Benchmarking Study, p. 22.
83 OAG, 2008 December Report: Chapter 2--Managing Severe Weather Warnings.
84 EC, 2007-2008 Report on Plans and Priorities, p. 39.
85 Kerry et al., Benchmarking Study, p. 22.
86 MSC, "Forecasts for Canadians, Foresight for Canada," p. 21.
87 Kerry et al., Benchmarking Study, p. 21.
88 Ibid., p. 21. Note: Amounts are presented in U.S. dollars.
89 Ibid., p. 21.
90 Ibid., p. 20.
91 Ibid., p. 20.
92 Ibid., p. 23.