This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Skip booklet index and go to page content

Evaluation of the Services to Marine Transportation Sub-activity

July 2011

Report Clearance Steps

Planning phase completed
November 2010
Report sent for management response
June 2011
Management response received
June 2011
Report completed
June 2011
Report approved by Departmental Evaluation Committee
July 2011

Acronyms used in the report

ADM
Assistant Deputy Minister
ASTD
Atmospheric Science and Technology Directorate
CCG
Canadian Coast Guard
CCRS
Canada Centre for Remote Sensing
CIOB
Chief Information Officer Branch
CIS
Canadian Ice Service
CSA
Canadian Space Agency
DFAIT
Department of Foreign Affairs and International Trade
DFO
Department of Fisheries and Oceans
DG
Director General
DND
Department of National Defence
EC
Environment Canada
FAR
Forecast Accuracy Ratio
IISPA
Ice Information Services Partnership Agreement
IMO
International Maritime Organization
ISO
International Organization for Standardization
JCOMM
Joint Technical Commission for Oceanography and Marine Meteorology
MCTS
Marine Communications and Traffic Services
METAREAs
Meteorological Areas
MSC
Meteorological Service of Canada
MOU
Memorandum of Understanding
NAIS
North American Ice Service
NASA
North American Space Agency
NOAA
National Oceanic and Atmospheric Administration
NRCan
Natural Resources Canada
OGDs
Other government departments
PAA
Program Activity Architecture
PMF
Performance Measurement Framework
POD
Probability of Detection
QMS
Quality Management System
SA
Sub-activity
S&T
Branch Science and Technology Branch
SMT
Services to Marine Transportation
SWG
Standards Working Group
TC
Transport Canada
WEPS
Weather and Environmental Prediction Services
WES
Weather and Environmental Services
WMO
World Meteorological Organization

Acknowledgements

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee as well as all interviewees and survey respondents who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was led by Gavin Lemieux, under the direction of the Environment Canada Evaluation Director, William Blois, and included Susan Wharton, Ryan Brown, Lindsay Fitzpatrick, Katheryne O’Connor, Kathryn Harris and Dan Wright.

Prepared by the Evaluation Division, Audit and Evaluation Branch

Return to Table of Contents

Introduction

Environment Canada’s (EC’s) Evaluation Division, Audit and Evaluation Branch, conducted the Evaluation of Services to Marine Transportation sub-activity between September 2010 and May 2011. This evaluation was conducted as part of a joint review and evaluation of the Services to Marine Transportation (SMT) sub-activity (SA),2 as per the 2010-2011 Departmental Annual Risk-Based Audit and Evaluation Plan, which was approved by the Deputy Minister in early spring 2010.

This project was identified by departmental senior management as an opportunity for testing the conduct of a joint review and evaluation. Although the project addressed two distinct lists of questions and adhered to two different sets of standards and procedures, efforts were made to conduct some audit and evaluation activities jointly in order to ensure a more streamlined data collection approach and approvals process, as well as a reduced burden on participating stakeholders.

This document presents the findings and recommendations of the evaluation component. Section 2 provides background and context information on the SMT activities being examined as part of this evaluation. Section 3 presents the evaluation design, including the purpose and scope of the evaluation, as well as the methods used to conduct the evaluation. Section 4 presents the evaluation’s findings. Section 5 and 6 lay out, respectively, the conclusions and recommendations. The management response is presented in Section 7.


2. The Program Activity Architecture (PAA) for EC portrays the Department's activities in a hierarchy showing how the lower–level elements (e.g., sub-activities or sub-sub-activities) ultimately contribute to the Department’s strategic outcomes. The entity being evaluated in this evaluation is a sub-activity, entitled 2.2.2 Services to Marine Transportation.

Return to Table of Contents

Background

2.1 Program Profile

The activities that take place under EC's SMT SA have a long history within the Department, but the introduction of the 2010-11 Program Activity Architecture (PAA) was the first time that they were identified as a unique element contributing to the Department's strategic outcomes.

EC's SMT sub-activity provides marine users with information and warnings to support decision making for marine activities and related threats to safety. It serves the needs of maritime communities, and supports Canada's obligations in providing safety information to international marine traffic. EC's SMT SA comprises two main activities: the provision of marine weather information and the provision of ice information.

Marine Weather Information

The provision of marine weather information includes marine forecasts and marine warnings. The marine forecast includes the provision of regularly issued short-term (Day 1 and 2) and extended (Day 3 to 5) extended marine forecasts, as well as several other products--including a wave height forecast and a recreational boating forecast, among others. Marine warnings include warnings of ongoing weather conditions and extreme weather events that pose a threat to life and property at sea, such as strong winds, freezing spray, high coastal waters, squall lines and other localized phenomena for the offshore economic zone, including the St. Lawrence Seaway and major inland waters. The criteria for issuing weather warnings are based on national standards, though they may be interpreted regionally to account for regional climatology.3 Providing this information involves the work of research, monitoring, prediction, and service delivery functions within the Meteorological Service of Canada (MSC), with support from the Science and Technology Branch (S&T Branch) and Chief Information Officer Branch (CIOB).

In addition to the atmospheric monitoring infrastructure employed in support of overall weather predictions, marine weather data also originate from a network of marine buoys, automated coastal stations, automated ship-borne weather stations, lighthouses, and volunteer observers aboard ships. Satellite data from the Canadian Space Agency (CSA) are used to monitor marine winds as well as ice and iceberg formation.4

Marine forecast and warning services are delivered through mass communication media such as radio, television, newspapers and the Internet, as well as targeted communications conduits such as Weatheradio, automated telephone answering devices, and the Continuous Marine Broadcast (provided by the Canadian Coast Guard [CCG]). There are also distribution relationships with national and regional media distributors and local emergency measures organizations.5

Ice Information

The second area of activity for the SMT SA is the provision of ice information, which is delivered as a partnership between the MSC's Canadian Ice Service (CIS) and the Department of Fisheries and Oceans (DFO) (specifically the CCG). The CIS is responsible for providing ice information, and the CCG is responsible for ice management. The CIS is structured into three units, responsible for: ice reconnaissance and field services; forecast operations; and applied science.6 The SMT SA supports the activities of the icebreaking program of the CCG, which include escorting ships through ice-covered waters, freeing vessels beset in ice, conducting harbour breakouts, providing advice and ice information, and reducing the risk of flooding on the St. Lawrence River through monitoring, prevention and breaking up of ice jams.7 The partnership between the CIS and the CCG is directed in part through a Memorandum of Understanding (MOU) and Partnership Agreement with DFO for services related to ice conditions over Canadian navigable waters.

Services under this activity support marine industries and other interests operating in Canadian waters, such as organizations involved in shipping, fisheries and resource extraction. Individuals and groups working in these industries are required to make tactical decisions (e.g., on ship routing) needed to maximize their effectiveness and safety. As a key collaborator, the CCG broadcasts ice information produced by ECto interested parties and provides in-situ weather and sea-state information to EC.

As described in the Partnership Agreement:

Information about ice conditions allows Canadians to make decisions concerning their activities in the vicinity of ice. Ice management with ice routing advice and icebreaker services allows ships to travel through heavy ice and prevents flooding as a result of ice jams. This program provides marine industries and regulatory agencies with forecasts of the sea state, ice conditions and weather, 24 hours per day, 365 days per year. Environment Canada delivers the information component of the program because of its scientific and technical expertise in developing environmental information, while the CCG delivers the management component because of its expertise in marine transportation and ship operations.8

EC is also a member of the North American Ice Service (NAIS). The NAIS is a collaborative undertaking between the MSC, the United States National Oceanic and Atmospheric Administration (NOAA), and the U.S. Coast Guard. The NAIS furthers navigation safety and informed decision-making through the provision of common ice–information products and services for North American and global waters.9Through this collaborative arrangement, the CIS,10 the National Ice Center11 and the International Ice Patrol12 are developing a common production system to develop a suite of harmonized ice information products and services to be shared between member organizations at no charge. Canadian contributions to this common production system come from the CIS Ice Service Integrated System.13

Legal and Statutory Commitments

Legal and statutory responsibilities for the SMT SA include the Department of the Environment Act and Oceans Act. The SMT SA also supports international commitments to the United Nations Convention on Safety of Lives at Sea, NAIS, NOAA, and World Meteorological Organization (WMO). International bodies in which the CISparticipates as a contributing member include the WMO Joint Commission on Oceanography and Marine Meteorology, and the International Ice Charting Working Group.14

2.2 Governance Structure

Within EC, the delivery of the SMTSA is performed by the MSC, the organizational branch with responsibility for producing and disseminating weather and environmental information, including that related to ice. The MSC also relies on the Atmospheric Science and Technology Directorate (ASTD) of the S&T Branch as a key contributor to meteorological research and development, and the CIOB for activities related to information management and information technology.

Management oversight of these activities is provided by the Weather and Environmental Services (WES) Board, a senior departmental committee composed of Assistant Deputy Ministers (ADMs) and Regional Directors General. Additionally, a new Marine and Ice Program Management Board has recently been formed. This board is composed of directors representing all groups delivering the marine transportation business line, and reports to the Director General of Weather and Environmental Prediction Services (WEPS).

Other committees and working groups supporting the delivery of the SMTSA include:

  • the Prediction Committee, which is responsible for cross-cutting operational program delivery issues;
  • the Prediction Centre Heads, who are the regional and national prediction managers responsible for delivering on operational requirements;
  • the Marine & Ice Programs Working Group, which is composed of representatives from national and regional offices delivering marine and ice services, including collaborators in monitoring, CIOB, dissemination and science; this working group oversees program changes and makes recommendations to the Marine and Ice Program Management Board;
  • the Standards Working Group (SWG), which coordinates the development, implementation and application of standards to ensure consistent delivery of the MSC's products and services, including standards related to International Organization for Standardization (ISO) requirements;
  • the Services Coordination and Management Committee, the senior committee to the SWG, coordinates the management of WES service activities, including coordinating responses to, and developing solutions of a national scope for, the marine weather program, as well as the programs for public weather, climate services, hydrometric information services, health sector services, economic sector services and commercial services.

2.3 Resource Allocation

The table below (Table 1) provides information on SMT SA expenditures from 2007–08 to 2010-11. The figures represent expenditures from the three branches involved in delivery of the SMT SA activities, i.e., the MSC, CIOB and the S&T Branch.

Table 1: Annual Financial Expenditures for 2.2.2 Services to Marine Transportation – 2007-2009 (text description)
Financial Expenditure2007-082008-09
MarineIceTotalMarineIceTotal
FTEs*319512632108140
Salary2,3862,8615,2472,6392,7715,410
O&M**1842,1192,3022122,6342,846
Capital077770337337
VNR***-Salary13,8073,80805,2005,200
VNR-O&M05,7165,71605,1195,119
TOTAL****2,57014,57917,1502,85116,06218,912

 

Table 1: Annual Financial Expenditures for 2.2.2 Services to Marine Transportation – 2009-2011 (text description)
Financial Expenditure2009-102010-11
MarineIceTotalMarineIceMet-areasTotal
FTEs*32108140321047144
Salary2,8563,0375,8933,0932,5886566,336
O&M**1241,5141,6381141,8692352,218
Capital01161160355177532
VNR***-Salary55,5885,59405,26105,261
VNR-O&M05,1165,11624,13824,142
TOTAL****2,98515,37218,3573,20914,2121,06918,491

* Full-time equivalents
** Operations and maintenance
*** Vote–netted revenue
**** Numbers are shown in $000s (except FTEs)

2.4 Regional Delivery of Services

Given the different regional needs (e.g., there is no need for a forecast for international mariners on Manitoba lakes), marine weather and ice information service delivery differs from region to region. The following table provides a high-level overview of the program's various services delivered in each EC region.

Table 2: SMT Service by Region15 (text description)
Program ServicesPacific CoastManitoba LakesHudson, Mackenzie and Arctic WatersGreat LakesSt. LawrenceAtlantic Coast
Short-term forecastXXXXXX
Weather warningsXXXXXX
Extended forecastXXXXXX
Technical marine synopsisX XXXX
Marine weather statementXXXXXX
Wave–height forecastX X16XXX
NAVTEX forecast (for international mariners in Canadian waters)X XXXX
Ice hazard bulletins  XXXX
Recreational boating marine forecast   X X
MAFOR forecast (internationally standardized version of the regular forecast, mainly for shipping industry)   XX 

2.5 Program Results Chain

Table 3 (below) is a representation of the marine transportation service activities and intended outputs and outcomes. This results chain was adapted from PAA element descriptions, the departmental Performance Measurement Framework (PMF) for 2010-11, and results from evaluation scoping interviews. It should be noted that the outputs and outcomes were developed for this evaluation by the ECevaluation team, because there was no finalized logic model and the PMF was in its initial phase prior to the evaluation. There was some concern expressed by SMTmanagement at the onset of the evaluation that the indicators may not be adequately aligned with SMT services and activities, and that management may not be able to collect the required data.

Table 3: Services to Marine Transportation Program Results Chain (text description)


Activities

  1. Ice research
  2. Marine weather warnings, forecasts and information
  3. Ice warnings, forecasts and information

Outputs

  1. Ice information products
    1. Ice charts and forecasts
    2. Warnings
    3. Regional charts
    4. Short- and long-term forecasts
    5. Iceberg bulletins and charts
  2. Ice information field services
    1. In situ ice information expertise
  3. Marine weather forecasts
  4. Marine weather warnings

Potential Reach / Target Clients

  • Industry: Shipping / marine transportation, commercial fishing, tourism, recreation, energy, construction
  • Federal Partners/Clients: DFO/CCG, Transport Canada (TC), Department of Foreign Affairs and International Trade (DFAIT), CSA
  • Provincial and Territorial Collaborators/Clients
  • Coastal Communities
  • International: United Nations Convention on the Safety of Lives at Sea, NAIS, NOAA, WMO, International Maritime Organization (IMO)

Outcomes

  • Increased access by target clients to accurate, timely and user-friendly weather and ice information products and services
  • Increased use of weather and ice information products and services by target clients to reduce their risks of operating in a hazardous environment
  • Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice
  • Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice

PAAOutcome: Element 2.2.2

Marine communities have the weather, wave and ice information they need to operate safely and efficiently in Canadian waters


PAAOutcome: Element 2.2

Targeted sectors routinely integrate weather and climate information into their operations


Departmental Strategic Outcome

Canadians are equipped to make informed decisions on changing weather, water and climate conditions



3. For example, the national standard for a gale is considered as 34 knots per hour or more, though there may be slight differences in interpretation across regions as to when the threshold of 34 knots is passed.

4. CSA. 2007. "RADARSAT Constellation."

5. EC. n.d. Environment Canada's Marine and Ice Warning and Forecast Programs. Internal document.

6. EC and DFO. 2005. Ice Information Services Partnership Agreement (IISPA) Between the Canadian Coast Guard Icebreaking Program of the Department of Fisheries and Oceans and the Canadian Ice Service of Environment Canada.

7. CCG. 2010. Levels of Service, p. 16.

8. EC and DFO. 2005. IISPA.

9. EC and the NOAA. 2007. Amendment 1 to the North American Ice Service: Annex 2 to the Memorandum of Understanding between the United States National Oceanic and Atmospheric Administration and the Meteorological Service of Canada of Environment Canada on Cooperation in Environmental Data Acquisition and Utilization.

10. Canadian Ice Service

11. National Ice Center

12. Navigation Center

13. North American Ice Service. 2009. Polaris Technology Collaboration Plan for the NAIS Common Production System, version 1.2.

14. EC and DFO. 2005. IISPA.

15. Data obtained from EC marine and ice information regional brochures.

16. Wave forecasts are provided for Eastern Hudson Bay and James Bay. Wave forecasts for the Arctic Waters, Mackenzie and Western Hudson Bay will be available in 2012.

Return to Table of Contents

Evaluation Design

3.1 Purpose and Scope

The purpose of this evaluation is to examine the relevance and performance of the SMT SA.17 This included investigating and reporting on:

  • The continued need for these services, and their alignment with government priorities (i.e., relevance);
  • The extent to which performance data are being collected and how well the SA is delivering on its outcomes, including the degree to which users' needs have been met (i.e., performance – achievement of outcomes);
  • The degree to which the SA is delivering its services in an economical and efficient manner (i.e., performance – demonstration of economy and efficiency).

The evaluation data collection phase began in November 2010 and continued until April 2011. During this period, there were periodic updates with representatives in the Audit and Evaluation Branch to share documents and preliminary analyses, and, where possible, conduct joint interviews.

In terms of scope, the SMT SA relies on a broad base of weather prediction services provided by other groups within the MSC and S&T Branch and supported by CIOB, as well as collaboration with other organizations. Although these activities provide important contextual information to better understand the SMT operations, they are not included within the scope of the evaluation per se.

Excluded from the scope of the evaluation are the services provided by the CIS relating to pollution monitoring, prevention and enforcement. These activities relate to the PAA element 3.1.2.2 – Marine Pollution.

Additionally, the new METAREAs (Meteorological Areas) Initiative in the Arctic, which involves the expansion of EC's domestic marine and ice services to provide a full suite of meteorological information for two of five new Arctic METAREAs established by the IMO, was being developed immediately prior to the evaluation data collection, and was therefore not sufficiently advanced in terms of outcomes to be included within the scope of this evaluation. Individuals did comment frequently on northern issues, however, and these are discussed in this evaluation as appropriate.

3.2 Evaluation Approach and Methodology

Four key considerations, outlined below, played a role in the design of the evaluation approach and methodology used for this evaluation.

First, as noted previously, this evaluation was initiated as a joint review and evaluation. This involved a joint planning phase with audit, including the initial risk assessment/scoping interviews being conducted by the audit and evaluation project leads, and presenting a joint review and evaluation plan to the responsible ADM for sign-off. Additionally, throughout the conduct of the project, opportunities to conduct a more streamlined data collection approach were undertaken where feasible, in an attempt to reduce the burden on participating stakeholders.

Second, this evaluation was a pilot for conducting evaluations at the PAA level. As such, methods were informed by the fact that this evaluation cut across two components within the PAA, with different stakeholders, program delivery methods and end-users.

Third, other recent evaluations have been conducted that offered insight into the SMT SA. This evaluation builds upon these previous evaluations, including relying on recent evidence as presented in these reports, to the degree that this was appropriate and feasible.

Finally, the evaluation took extra efforts to understand the perspective of end-users of EC's services to marine transportation, because it is required in order to adequately assess the degree to which the SMT SA is achieving its outcome of ensuring that marine communities have the weather, wave and ice information that they need.

With these considerations in mind, the following three methods were used in the conduct of this evaluation, ensuring that each evaluation question was addressed with multiple lines of evidence:

  • Document review – The evaluation team requested all pertinent program documents, such as program planning and legislative documents, Acts of Parliament, minutes from EC Board meetings and performance measurement data collected by the program. To facilitate document review, members of the evaluation team were granted access to the SMT shared drive. Financial data were also obtained from EC's Financial Management Branch. EC's Evaluation of Research and Development, Production and Monitoring in Support of Weather Prediction,18 and DFO's evaluation of its Icebreaking Program19 were also reviewed. Documents received by either the evaluation or the audit team were shared with the other.
     
  • Open-ended interviews – A long-form interview guide (see Annex B) was administered to 29 individuals (detailed below), including those involved in the delivery of SMT as well as end-users of its services. Interviewees who participated in both the review and evaluation components were provided with the opportunity to conduct one joint interview or two separate interviews. The purpose of the open-ended interviews was to engage individuals that were knowledgeable about the SMT SA's activities and impacts based on their professional association with EC's SMT SA services. Key informants representing the marine weather information and ice information components of the SA were interviewed. In the majority of cases, the key informant was knowledgeable about one of these two components and few were able to speak to the SA as a whole. Table 4 provides a detailed breakdown of key informant interviewees.
    Table 4: Key Informant Interviewees: Long-form Interview Guide (text description)
    Key Informant TypeNumber Interviewed
    MSC Policy and Planning3
    Marine Program Management7
    Ice Program Management1
    Marine Services4
    Ice Services3
    Regional Operations (Marine and Ice)4
    Clients (Other Government Departments)3
    End-Users (Industry and Shipping)4
    Total29
  • Questionnaire for End-Users – A questionnaire was administered to end-users of SMT SA services. The purpose of the questionnaire was to obtain information on the use of and satisfaction with SMT SA services among mariners who were the target clients of these services. The evaluation team was provided with a list of key industry and non-governmental organization contacts. These individuals were contacted by telephone initially and, if willing to participate, were sent an email providing greater detail on the purpose of the evaluation, along with a copy of the guide in both official languages. Participants had the option of either filling out responses directly on the paper copy and faxing it back to EC Audit and Evaluation Branch officials, or arranging for an interview with a branch official. The vast majority (27 out of 33) chose to fill out the guide and fax or mail their responses back to EC. Table 5 indicates the distribution of respondents to the end-user questionnaire.
     
    Table 5: End-user Type: Questionnaire (text description)
    End-user TypeNumber Interviewed
    Recreational mariner / sport fisher7
    Professional mariner2
    Professional fish harvester18
    Ship manager or operator3
    Other3
    Total33

3.3 Limitations of the Evaluation

The SMT program is a new entity in the 2010-11 PAA, in which all service activities targeting marine transportation have been grouped under a common results structure. As such, this program entity does not yet benefit from a fully articulated program logic and clear governance structure.20 In order to address this, the evaluation team developed a logic model with activities, outputs and outcomes for the purpose of the evaluation, which may not represent the final indicators selected to measure performance for the SA. The evaluation team made efforts, however, to ensure that the logic model was an effective tool for assessing performance for the SA, by basing it on existing performance documents and validating it with the evaluation steering committee and senior management. Furthermore, where available and relevant, additional performance data beyond measures for the identified outputs and outcomes were also assessed.

Although the number of respondents to the questionnaire exceeded the evaluation team's original goal of 30, the results of the end-user questionnaire cannot be viewed as a statistically valid representation of SMT end-users. The sample size was too small to provide statistical validity, and it was skewed toward the views of the east coast fishing industry, with almost two thirds of respondents coming from this group. This over-representation appears to be because east coast fishing organizations contacted for the evaluation were extremely effective in distributing the questionnaire to their membership. The findings from this line of evidence, therefore, should be viewed as illustrative, and treated with caution. To compensate for this, the evaluation also relied on other sources of evidence, including the results of a previously conducted survey of end-users.


17. An evaluation matrix, including evaluation questions, indicators and methods/sources, is included in Annex A.

18. EC, Audit and Evaluation Branch. January 2011. Evaluation of Research and Development, Production and Monitoring in Support of Weather Prediction.

19. DFO. April 2011. Evaluation of the Icebreaking Program.

20. Responsible managers on the Marine and Ice Program Management Board are, as of the publication of this report, undertaking the development of a program logic model, performance measures and clear governance structure specific to the SMT SA.

Return to Table of Contents

Findings

The findings of this evaluation are presented below by evaluation issue (relevance and performance) and the related evaluation questions. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question based on a judgement of the evaluation findings. The rating symbols and their significance are outlined below in Table 6. A summary of ratings for the evaluation issues and questions is presented in Annex 2.

Table 6: Rating Symbols and Significance (text description)
SymbolSignificance
AchievedThe intended outcomes or goals have been achieved or met
Progress Made; Attention NeededConsiderable progress has been made to meet the intended outcomes or goals, but attention is still needed
Little Progress; Priority for AttentionLittle progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis
N/AA rating is not applicable
~Outcomes achievement ratings are based solely on subjective evidence.

4.1 Relevance

Evaluation Issue: Relevance


Overall Findings: Relevance

Multiple sources indicate that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions. Findings from multiple sources also indicate that activities and services offered under the SMT SA are consistent with the legislated mandates of the involved departments and with responsibilities assigned through agreements and other mechanisms for collaboration. The need for SMT services was particularly apparent given the evidence indicating that the MSC is obligated by various acts and agreements to deliver SMT SA services, including in the context of ongoing northern development. In addition, it was noted by key informants that the SMT SA is aligned with and supports federal government priorities and departmental strategic outcomes.

Some concerns were expressed regarding the extent to which some areas of shared responsibility are clearly articulated. Furthermore, there is additional concern that the Department's ability to deliver on its responsibilities is vulnerable, as these services are reliant on key collaborating departments and agencies.


4.1.1 Continued Need for the Program

Continued Need for the Program
Evaluation Issue: RelevanceIndicator(s)MethodsRating
1. Is there a continued need for SMT?
  • Evidence of / views on societal/ environmental need for program
  • Document review
  • Literature review
  • Key informant interviews
Achieved

Multiple sources indicate that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions. The continued need is particularly apparent given that the MSC is obligated by various acts and agreements to deliver SMT services, including in the context of ongoing northern development. This need was also recognized by all key informants interviewed as part of the evaluation, with some indicating that it is fundamental to supporting the health and safety of mariners, instrumental in supporting fishing and other marine industries, and important in the context of northern climate change and economic development, as the level of maritime traffic in the north has been increasing due to longer navigation periods in Arctic waters.

The following extract outlines the key reasons for the provision of ice and marine weather information as a public service:

As a northern nation, Canada must know about its ice environment and the impacts it has on human activities. As a trading nation dependent on marine transportation, we must deal with the hazards that the constantly shifting ice presents to ships. Three of the five largest Canadian ports are affected by ice for a large part of the year and rely on ice information to stay active and viable. It is believed that because of the widespread public impact that ice has on broad sectors of the economy and because of the safety hazards it presents to the marine public, there has been little question that providing ice information is an essential federal government service. The public benefits are too widely dispersed to sustain a viable private sector service except in certain niche areas, such as site–specific information for offshore oil platforms. Like all other northern nations, Canada provides an ice information service as a public benefit.21

The need for services offered by SMT is expected to continue and even increase. The CCG Fleet Annual Report for 2007-2008 noted that "Climate change, notably in the Arctic, is already extending the duration of the commercial shipping season, and may further intensify the demand for marine science and other activities."22

The document review indicates that the provision of SMT SA services should not be considered an optional service for EC. Rather, EC is obligated by a variety of acts and agreements to provide these services, both in partnership with other departments and directly to Canadians. For example23:

  • Canada is a signatory to the United Nations Convention on the Law of the Sea (UNCLOS). Article 43 of UNCLOS charges states adjoining straits to cooperate in aiding navigation and in preventing, reducing and controlling pollution from ships. The provision of SMT thus enables Canada to meet its international obligations respecting marine safety.24 Similarly, Canada is a signatory to the International Convention for the Safety of Life at Sea. Chapter V of this convention lists certain navigation safety services that should be provided by governments, including the maintenance of meteorological services for ships, the ice patrol service, routing of ships, and the maintenance of search and rescue services.25
  • Canada has committed to the IMO to provide meteorological information and navigational data to facilitate the safe management of marine traffic in two well-defined Arctic areas that are substantially within Canadian territory. The areas include Canadian Arctic waters, such as the Northwest Passage, as well as adjacent waters north of Alaska and along part of the western coast of Greenland.
  • As noted in the Department of the Environment Act, the Minister of the Environment has legislative authority for all matters relating to meteorology, water, and boundary waters, and for the provision of environmental information to Canadians.26 As such, EC is federally mandated to provide authoritative, quality meteorological and environmental warnings, forecasts, services and information for the overall betterment of Canada.27
  • Since 1995, the CIShas entered into consecutive five–year MOUs with the CCG to provide information related to ice conditions in order to assist that agency in meeting its mandate.
  • A Senate Report on the CCG in 2009 (and by implication the CIS) recommended that Canada develop a much stronger year-round national presence and enforcement capability, to demonstrate commitment to controlling the Northwest Passage, protecting Canadian interests and Canada's northern residents, and making the waterway a safe and efficient shipping route.28
  • Budget 2010 provides $9.2 million over two years to EC and $2.2 million over two years to DFO to deliver meteorological and navigational services, respectively, in the Arctic, in order to meet Canada's commitments to the IMO regarding these areas.29

The continued need for the services of the SMT SA was also recognized by all key informants interviewed as part of the evaluation. A plurality of respondents indicated that they thought the SMT SA is fundamental to supporting the health and safety of mariners, and that it is instrumental in supporting the fishing and other marine industries. Furthermore, approximately one in four key informants indicated that there are no alternative providers of these SMT SA services and that these services are a public good. Approximately one in four indicated that the SMTSA is particularly important in the context of northern climate change / economic development, as the level of maritime traffic in the north has been increasing due to longer navigation periods in Arctic waters. Other comments included the fact that the SMT SA is essential for the activities of other federal departments (examples include supporting the CCG and the Department of National Defence [DND] in search and rescue operations and environmental protection), and that SMT SA supports the development of private sector organizations that use generalized EC data to provide a level of detailed information (e.g., for offshore drilling and support to northern communities).

4.1.2 Alignment with Federal Government Priorities

Alignment with Federal Government Priorities
Evaluation Issue: RelevanceIndicator(s)MethodsRating
2. Are Services to Marine Transportation aligned with federal government priorities?
  • Program objectives correspond to recent/current federal government priorities
  • Program objectives are aligned with current departmental strategic outcomes
  • Document review
Achieved

The SMT SA is aligned with and supports federal government priorities (e.g., health and safety, the economy, infrastructure and sovereignty) as well as departmental strategic outcomes and priorities.

Alignment with Federal Government Priorities

In terms of recent priorities, the Government of Canada has been active in promoting and defending Canada's sovereignty over Arctic waters. The 2010 Speech from the Throne contained the following commitment:

Our Government will continue to vigorously defend Canada's Arctic sovereignty. It will continue to map our northern resources and waters. It will take action to increase marine safety and reduce pollution from shipping and other maritime traffic.30

Arctic sovereignty was a theme of the 2007 Speech from the Throne as well, which highlighted a need for improved patrol, surveillance, and environmental research capabilities.31SMT services such as supporting patrol and surveillance capacity are essential to furthering the goals of establishing and defending sovereignty, and to the ongoing development of Canada's North.

Budget 2010 contained several provisions related to the SMTSA. Most directly, the Budget provided $9.2 million to ECand $2.2 million to DFO, over two years, to deliver meteorological services in the Arctic to help meet Canada's commitments to the IMO. Other Budget provisions that indicate the continued importance of SMT include $497 million over five years for the CSA's RADARSAT Constellation Mission (which has marine and ice applications); and $27.3 million over five years to aid the CCGin ensuring maritime safety, as well as $175 million to the CCG for the repair and refit of existing vessels and the purchase of new vessels.32

Key informants also indicated that the SMT SA is aligned with government priorities. Nearly half of the key informants indicated that the SMT SA helps support the Canadian economy and efforts for economic development (particularly in the North), and that the SMTSA contributes in key ways to ensuring the health and safety of Canadians. Several key informants also noted that the SMTSA is important for federal efforts to enhance infrastructure in the North.

Alignment with Departmental Strategic Outcomes

SMTinformation products (e.g., ice and marine weather warnings) are designed to help people across Canada respond appropriately to marine and ice weather conditions. The newly developed Marine and Ice Program Management Board (MIPMB) is tasked with managing the activities of EC's SMT as identified within EC's PAA.33,34 The delivery of the SMTSA supports the departmental strategic outcome "Canadians are equipped to make informed decisions on changing weather, water and climate conditions." Weather prediction activities also align with one of three strategic outcomes in EC's Sustainable Development Strategy 2007-2009: "Weather and Environmental Services: Weather and environmental predictions and services reduce risks and contribute to the well-being of Canadians."35

Furthermore, key informants also noted that SMT is aligned with departmental priorities. For example, approximately one in five key informants indicated that the SMT SA helps support departmental efforts to protect the environment, such as from oil spills to water bodies, and that it contributes to the safety and security of marine transport. Several key informants also indicated that the SMTSA helps support the integrated decision making of EC's WESBoard.

4.1.3 Consistency with Federal and Departmental Roles and Responsibilities

Consistency with Federal and Departmental Roles and Responsibilities
Evaluation Issue: RelevanceIndicator(s)MethodsRating
3. Are Services to Marine Transportation consistent with federal and departmental roles and responsibilities?
  • Program mandate aligned with federal government jurisdiction
  • Evidence of / views on legitimacy, appropriateness and necessity of departmental roles, responsibilities and accountabilities with respect to marine transportation
  • Evidence of duplication and/or complementary activities with other government departments (OGDs)
  • Document review
  • Key informant interviews
Achieved

The provision of the SMT SA is consistent with the legislated mandates of the involved departments and with responsibilities assigned through agreements. For the most part, comments from key informants focused on the SMT SA services' complementary nature with OGDs. Some concerns were expressed by EC and collaborating organization representatives regarding the extent to which some areas of shared responsibility are clearly articulated.

The roles and responsibilities for the provision of services to marine communities and marine industries are spread across EC, DFO and TC. Generally, the commitments of the various departments with respect to providing services to aid marine transportation are as follows:

  • EC has a responsibility to provide meteorological information of interest to Canadians, for general and emergency management purposes.
  • DFO has a responsibility to provide ice management, channel maintenance, and navigational information. Although these services may be augmented by the assistance of other departments, DFO is authorized to conduct the research and information production activities that are necessary to meet its requirements.
  • TC has a responsibility to regulate marine transportation activities in order to promote the health and well-being of those involved in marine transportation.

EC's mandate for federal involvement in providing services to aid marine transportation is set by sections 4 and 5 of the Department of the Environment Act, which assigns to the Minister of the Environment jurisdiction over water and meteorology issues and responses, as well as the provision of environmental information to Canadians and the development and coordination of environmental programs.

Section 6 of the Emergency Management Act requires all departments to develop and implement emergency management plans in relation to the risks associated with their area of responsibility. In terms of the coordination of efforts between levels of government in the case of meteorological emergencies, EC is responsible for issuing marine warnings; flood warnings are issued by provincial governments.36

The following table sets out the various linkages with other organizations, for EC's SMT, by subject area.

Table 7: SMT Linkages by Subject Area37 (text description)
Subject AreaLinkages
Marine safety and navigation in iceTC, International Hydrographic Organization, IMO, Shipping Federation of Canada, U.S. Coast Guard, International (North Atlantic) Ice Patrol, International Navigation Association
Marine meteorology and ice information servicesWMO/JCOMM Expert Team on Sea Ice, U.S. National Ice Center, other national ice services, International Ice Charting Working Group, others
Satellite missions for ice and marine surveillanceCSA, European Space Agency, Natural Resources Canada (NRCan)–Canada Centre for Remote Sensing (CCRS), NOAA, North American Space Agency (NASA), Japanese Space Agency
Satellite data processing and satellite-based operational servicesCSA, European Space Agency, NRCan–CCRS, NOAA, NASA
Arctic transportation and sovereigntyDFAIT
Ice climatology and climate change researchClimate Research Branch–MSC, Canadian Foundation for Climate and Atmospheric Sciences, Canadian Collision Industry Forum, National Snow and Ice Data Center, ArcticNet, Canadian Arctic Shelf Exchange Study
Ice engineering and environmental designNational Research Council–Canadian Hurricane Centre, C-CORE, other industry

For the most part, comments from key informants focused on the complementary nature of SMT SA services with OGDs. For example, nearly half of the key informants noted that the SMT provided by EC do not duplicate private sector endeavours. Private sector endeavours tend to build upon publicly provided information, and are frequently aligned with a specific set of users (e.g., skiers, offshore oil and gas industry). Approximately one in four also noted that the SMTSA complements and supports the work of other organizations (e.g., DFO/CCG, TC, International Joint Commission, CSA, U.S. National Ice Service).

However, key informants pointed to the potential risks associated with the extent to which the SMT SA relies on collaborators to meet its program objectives (e.g., DFO and NAV Canada). The risk exists that, if these relationships were terminated, EC may be unable to deliver on certain services until other suitable collaborators could be identified. If business relationships with collaborators or third parties are not well managed, or if there is a failure on the part of collaborators, this could negatively affect program or service delivery. This issue is discussed further in the section related to efficiency and economy.

4.2 Performance

Evaluation Issue:


Overall Findings: Performance

Overall, the document review and key informant interviews indicate that there has been notable effort to ensure that the performance measurement data collection and reporting system provides information on key outputs and outcomes, and that these are currently managed through the early PMF development and ISO 9001 Quality Objectives. The main area for improvement at present would appear to be developing comprehensively managed reporting through methods such as an annual report, in order to ensure that the wealth of data are reported in a consistent and unified manner. There is also some room for improvement with respect to the completeness and regional consistency of data and the use of performance data to inform senior management / decision makers.

A review of the evidence found that the SMT SA is able to provide data and targets on its key outputs, and that the SMT SA is performing at or near the vast majority of these targets. With regards to performance against outcomes, evidence of strong performance regarding access to and use of weather and ice information products exists. Data were more limited for the outcomes related to a reduced need for emergency interventions, and reduced unexpected delays by commercial marine transports due to marine weather or ice, but some information was available nonetheless. However, given the recent establishment of the SA and outcomes, no historic data exist to measure the degree to which these outcomes have seen changes over time. Data gathered during this evaluation can be used for future refinement of a PMF and, where appropriate, as a baseline against which future performance can be measured.

The evidence related to efficiency and economy is preliminary, but points toward an SA that is taking steps to ensure that SMT SA inputs are appropriate and costs of outputs are minimized. Areas for improvement and future challenges include clarifying roles and responsibilities with DFO/CCG related to servicing buoys and storm surges, and exploring possible cost savings opportunities associated with a better use of emerging technologies.


4.2.1 Collection and Application of Performance Data

Collection and Application of Performance Data
Evaluation Issue: PerformanceIndicator(s)MethodsRating
4. Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?
  • Existence of effective performance measurement strategy:
    • Performance indicators cover all outcomes
    • Performance indicators are linked to the outcomes being measured
    • Evidence of rigorous performance data collection and reporting activities
    • Satisfaction of decision makers with timeliness of data collection/reporting
    • Evidence of fully populated database(s)
  • Evidence of reporting of performance data to senior management / decision makers
  • Document review
  • Key informant interviews
  • Performance data analysis
Progress Made; Attention Needed

Overall, the document review and key informant interviews indicate that there has been notable effort to ensure that the performance measurement data collection and reporting system provides information on key outputs and outcomes, and that these are currently managed through the draft PMF and ISO 9001 Quality Objectives. The SMT SA is able to provide data and targets on its key outputs; and performance data exist, although in a more limited manner, on its identified outcomes. The main area for improvement is developing comprehensively managed reporting in order to ensure that the wealth of data being collected is reported in a consistent and unified manner, so as to inform senior management and decision makers. There is also some room for improvement with respect to the completeness and regional consistency of data.

The SMTSA has many of the key components for the development of an effective PMF, and evidence shows that attention has been given to this area. At a broad level, the evaluation found that the SMT SA has been generally effective at collecting performance measurement data. The level of available data is perhaps not surprising, given that the main program components of the SMTSA have been in place for many years. There are regular performance reports for the CIS, CCG icebreaking, and public and marine warnings, and the CIS/CCG partnership agreement outlines performance measures. Furthermore, there was a PMF under development during the evaluation to specifically address the SMT SA, as well as performance measures established in the Quality Management System (QMS) Manual (as part of the ISO 9001 Quality Objectives),38 and a Marine Forecast Verification Project aimed at improving the quality of marine forecasts and warnings. Details are as follows:

  • A draft PMF for the SMT SA for 2010-2011 outlines a framework for measuring several outputs and outcomes, although targets and timelines had not been established for all of the indicators.39
  • The Ice Information Services Partnership Agreement (IISPA) sets out performance measures for the CIS's provision of services to the CCG.40 The performance measures set out in this agreement are collected, for the most part, in the Canadian Ice Service Monthly Performance Measurement Report, which delivers on the agreements' intended monthly and quarterly reports. However, there were a few measures missing from the report, and several related to quality objectives that were supposed to be collected through an annual survey. Data that were to be collected through this annual survey do not appear to exist, due to restrictions on survey work that could be classified as public opinion research.
  • The WESQMS Manual indicates that efforts are being made to develop and track performance measures for: marine warnings, forecasts, and information products; and ice information products (ice and iceberg forecasts, warnings, and charts) as well as ice service specialists.41
  • The Marine Forecast Verification Project shows that efforts are being made to develop a nationally consistent program that verifies the performance of marine forecasts and marine warnings across Canada, to complete the performance measurement initiatives associated with all of its primary forecast programs. It will "fill an information gap that has existed for some time" and "ensure that a detailed database of forecast and observational data is maintained.42

The evaluation found that data are available to some degree for all of the outputs and outcomes identified in the draft PMF43 for the SA and those identified in the evaluation plan.44,45 Most of the performance measures appropriately capture the outcomes that are being measured, although in some cases the data do not align precisely or the periods of coverage are inadequate or have gaps. Table 8 summarizes the data collected against the identified outputs and outcomes.

Table 8: Performance Measurement Data Availability (text description)
IndicatorsData collectedAvailability
Output – Ice information products: Percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines
  • Percentage of ice forecast bulletins that were delivered on-time by region
  • Percentage of ice charts that were delivered on-time by region46
– Monthly data
Percentage of ice warnings issued and verified prior to the event
  • No data
– No data
Output – Ice information field services: Frequency (percentage of times) that trained ice service specialists are provided as per partnership agreements
  • Person days delivered as a percentage of person days requested47,48
– Monthly data
Output – Marine weather information products: Percentage of gale, storm, and hurricane-force warnings issued at least 24 hours in advance of the event
  • Percentage of gale, storm, and hurricane-force warnings with lead times of 24 hours or more49
– Quarterly data
Evidence of forecast accuracy
  • Probability of Detection (POD) – gale, storm, and hurricane force50
  • Forecast Accuracy Ratio (FAR) – gale, storm, and hurricane force51
  • Critical Success Index – gale, storm, and hurricane force52
– Quarterly national data and monthly regional data
Outcome – Access to and use of meteorological, oceanographic, and ice information products and services by target users
  • Percentage of mariners using marine weather forecasts, and daily frequency of use
  • Frequency with which mariners change sailing plans due to forecasts and percentage following recommended ice route
  • Use and source of weather information
  • Frequency of use of Port Meteorological Officer services, radio aids for marine navigation, and Navigational Telex (NAVTEX) and Marine Forecast (MAFOR)
– Survey conducted in 2004, skewed toward recreational mariners53,54
Outcome – Access to and use of meteorological, oceanographic, and ice information products and services by target users
  • Icebreaking services requested by region, country, service requested, vessel and cargo
  • Weatheroffice website and Really Simple Syndication visits for marine products

– Annual data55

– Annual data56

Outcome – Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice
  • Number of shipping accidents classified as ice damage
– 21 years of annual data are available from the Transportation Safety Board57
Outcome – Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice
  • Client delay due to ice conditions as a percentage of total service delays
– Annual data58
PAA Outcome – Marine communities have the weather, wave and ice information they need to operate safely and efficiently in Canadian waters: Target end-users' satisfaction with the accuracy, timeliness, accessibility / user-friendliness and usefulness of marine weather and ice information products and services
  • Level of understanding, usefulness, accessibility, timeliness and accuracy of weather information and reasons for low ratings
  • Importance of forecasts
  • Importance of daily ice charts and recommended ice route
  • Percentage of mariners reporting that the weather elements warning system meets needs
– Data from a 2004 survey59
PAA Outcome – Marine communities have the weather, wave and ice information they need to operate safely and efficiently in Canadian waters: Target end-users' satisfaction with the accuracy, timeliness, accessibility / user-friendliness and usefulness of marine weather and ice information products and services
  • Client engagement sessions concerning clients' issues and concerns with services
  • Received feedback during the Maritimes Region Marine Day from discussions and a survey
  • Received feedback during the North Atlantic Fish and Workboat Show from discussions and a survey

– 30 regional client engagement sessions in 200760

– Discussions with 80 visitors, and 27 surveys61

– Discussions with 280 visitors, and 59 surveys62

The key informant interview responses on the state of performance measurement for the SA generally align with the findings from the review of documents related to performance measurement. Generally, the opinion from key informants was that performance measurement has recently undergone, and is continuing to undergo, substantial improvements. Several individuals indicated that the ISO 9001 Quality Objectives were an important contributor to improvements in performance measurement. The Marine Verification Project was also noted as a contributor to improved quality of performance measurement information, as it collects data on the accuracy of marine weather products to support ongoing improvements.

Though the majority of interviewees suggested that performance measurement data are being used to inform decision–making processes, no documents were found in the document review that indicated decisions were being made based on collected performance measurement data. During the interviews, however, key informants did provide specific examples, including the following:

  • The Marine and Ice Program Board applies performance measurement information from end-users to guide priorities for operations (for example, extending the forecast period to five days).
  • Data are being used to improve forecasting. As data for verification of products such as wind warnings are collected and analyzed, they are used to provide feedback to forecasters, and if an obvious missed forecast has occurred, there may be a case study with regard to the event.
Examples of areas for improvement

Several areas of improvement were noted by key informants:

  • Five of the 29 key informants noted a general desire for more data to better understand users' needs. Restrictions on the conduct of public opinion research and the expense associated with running focus groups has limited this work, but these key informants noted that the SMT SA needs to improve how it obtains client feedback, to better understand if they are meeting client needs. It was noted, however, that efforts are being made to improve how feedback obtained from end-users by the National Inquiry Response Team is shared within the organization.
  • Five key informants noted that the SMT SA is working toward developing a verification system locally and nationally, but would like to see the national verification system developed more vigorously. Two of these key informants noted that there may be more national work carried out in the next six to twelve months and that a new national system and working group should enhance information–sharing among regions.
  • Four key informants noted that the SMT SA is strong in terms of collected data at the output level (e.g., related to issuance of warnings and forecasts), but not as strong in terms of tracking impacts at the outcome level.
  • Three key informants noted that they have not yet seen systematic performance data come forward to the Marine and Ice Program Board, and it was noted that, in spite of the "huge leaps forward" in terms of EC's ability to predict accurately for the marine community, there is not enough data to demonstrate this progress.

A 2011 DFO-led evaluation of the CCG's Icebreaking Program63 noted that CCG management had concerns regarding the level of performance reporting from the CIS. At the time of that evaluation, CCG representatives identified that delayed performance reporting from the CIS limited the extent to which the Icebreaking Program could report to Parliament on the full range of activities and outcomes. However, during this evaluation, CCG representatives noted that the CIS had greatly improved performance reporting in recent months.

4.2.2 Achievement of Outputs and Outcomes

Achievement of Outputs and Outcomes
Evaluation Issue: PerformanceIndicator(s)MethodsRating
5. To what extent have intended outcomes been achieved as a result of the program?

Evidence of / views on achievement of intended outputs and outcomes

Outputs:

  • Percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines
  • Percentage of gale warnings issued at least 24 hours in advance of the event
  • Percentage of ice warnings issued and verified prior to the event
  • Frequency (percentage of times) that trained ice service specialists are provided as per partnership agreements

Outcomes:

  • Increased access by target clients to accurate, timely and user-friendly weather and ice information products and services
  • Increased use of weather and ice information products and services by target clients, to reduce their risks of operating in a hazardous environment
  • Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice
  • Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice

Evidence of / views on factors outside the direct purview of the program that may influence / have influenced the achievement of intended outcomes

Document review

Key informant interviews

Performance data analysis

Achieved

A review of the evidence, including evidence based on activities prior to the formal integration of the weather and ice services under the SMT SA in 2010–11, found that the SMT SA is performing at or near many of its targets for its key outputs, such as the delivery of ice forecasts, charts and marine weather warnings. Furthermore, the SMT SA appears to be achieving three of the four listed outcomes, including demonstrating access to and use of SMT products and services by key clients. The evaluation evidence also pointed to an impact on the reduction of emergency interventions, though the evidence was indirect. The fourth outcome, related to reduced delays for maritime transportation, may not be feasible to measure at present. Given the recent establishment of the SMT SA and its associated outcomes, no historic data exist to measure the degree to which these outcomes have seen changes over time.

Achievement of Outputs

Generally speaking, the SMT SA has been performing at or near the vast majority of established targets for program outputs. More specifically, evidence collected from the SMT SA indicates that forecast bulletins have been meeting their performance targets and that ice charts have been just below the stated target64. Furthermore, the provision of ice service specialists has been consistently meeting the performance target.65

Gale warning lead times, and storm and hurricane–force warnings, are not yet performing at the targeted level; this is partly corroborated by mariners' ratings of the timeliness of marine warnings.66 The POD and FAR for storm and hurricane-force warnings have been performing well, while gale force warnings have been somewhat weaker in their performance relative to the stated target.67 More detail regarding these outputs and results is provided in Table 9.

Table 9: Detailed list of Outputs and Results(text description)
Data collectedResults

Ice information products

  • Percentage of ice forecast bulletins that were delivered on-time by region
  • Percentage of ice charts that were delivered on-time by region
  • With ice forecast bulletins delivered on time 96% of the time, SMT is meeting its performance target, as set out in the 2010–2011 PMF, of 100% of products transmitted within 30 minutes, 95% of the time.
  • With ice charts delivered on time 92% of the time, SMT is very close to but not yet meeting its target, as set out in the 2010–2011 PMF, of 100% of products transmitted within 30 minutes, 95% of the time. However, the PMF2010–2011, which set out the target of 95%, set to achieve this target in 3 years.68

Ice information field services

  • Person days delivered as a percentage of person days requested for ice service specialists
  • The person days delivered as a percentage of person days requested averaged 105%, which indicates that ice service specialists are being delivered as per partnership agreements, and thus meeting the 2010–2011 PMF goal of trained ice service specialists being provided 100% of the time.69

Marine weather information products

  • Percentage of gale, storm, and hurricane–force warnings with lead times of 24 hours or more
  • With lead times of 24 hours for marine gale warnings approximately 50% of the time, SMT is not meeting its performance target, as set out in the 2010–2011 PMF, of 80%. However, a date for achievement of this target had not been established in the PMF. In addition, documentation indicates that the existing PMF target was to be revisited to ensure that the targets are realistic.70 No targets had been set for storm or hurricane-force winds, but their lead times were even lower, at 35% and 10%, respectively.71
  • POD and FAR for gale, storm, and hurricane-force warnings
  • Gale warnings are consistently meeting the POD target of .6 or greater, and are meeting the FAR target of .3 or less in the majority of the quarterly periods. Storm warnings are performing similarly to gale warnings. Though there are data on the performance of the hurricane–force warnings, given the rarity of hurricanes the data were not considered sufficiently robust to report as evidence.72
Achievement of Outcomes

At this point, although there was ample quantitative and qualitative evidence to examine each of the proposed indicators, given the lack of historic reporting for the SA the evaluation team was unable to identify changes over time in the indicators that could quantify actual increases.

Outcome 1: Increased access by target clients (OGDs and end-users, both domestic and international) to accurate, timely and user–friendly weather and ice information products and services

There is a range of evidence suggesting that meteorological, oceanographic and ice information products and services are being accessed by the targeted users. Survey data collected by the MSC indicated that a large majority of mariners use marine weather information, and almost half frequently change their plans based on this information, while two thirds of mariners indicated that they usually followed the recommended ice routes. Mariners are very satisfied with the timeliness, accessibility / user-friendliness and usefulness, but somewhat less satisfied with the accuracy, of marine products. A large majority of participants found weather information products and ice products to be important and that the warning system was meeting their needs.73

Another indicator of access is the number of website hits. In recent years, there have been, on average, over 30 million visits annually to the marine weather portion of the Weatheroffice website and over 1.5 million visits on the Really Simple Syndication feed for marine products.74 Furthermore, surveys indicate that the EC Weatheroffice website is one of the most frequently used sources of marine weather information by mariners. These surveys also indicated that the CCG Continuous Marine Broadcast was another frequently used source of marine weather information.75,76 The CCG obtains its information for the broadcast from EC.

Figure 1 presents the distribution of marine weather sources, based on the survey of end-users. It shows both the general sources of maritime weather information (i.e., all sources accessed by end-users) and the main source of maritime weather information (i.e., the primary source accessed by end-users). These survey findings support the results on access noted above, as 27 of the 33 respondents (82%) indicated that the EC Weatheroffice websiteis one of their sources for marine weather information, and 19 (58%) indicated that the site is their main source of weather information. After the Weatheroffice website, sources such as other Internet sites, Weatheradio, the continuous marine band frequency, the marine radio frequency, radio, television and the CCG's communications system were all highly cited as sources of weather information. The Weatheroffice website, however, was the most–cited mainsource of information.

Figure 1: Distribution of Marine Weather Sources Used by Respondents (text description)

Figure 1: Distribution of Marine Weather Sources Used by Respondents

Feedback received from attendees at two marine events attended by EC representatives identified an access issue with Weatheradio broadcasts: attendees remarked that because these broadcasts alternate between separate English and French broadcasts at fixed intervals, longer broadcasts are frequently cut off before completion, as the broadcast must switch to languages at the scheduled time, thus posing a safety risk and limiting the utility of the product for mariners.77,78 The issue of the alternating broadcast was also highlighted by about half of the key informants interviewed (including individuals external to government), by one respondent in the survey of end-users, and through a review of program documents.79

Outcome 2: Increased use of weather and ice information products and services by target clients (OGDs and end-users, both domestic and international) to reduce their risks of operating in a hazardous environment

As with access to marine forecast products, the evaluation team found a range of evidence to suggest that SMT SA products and services are being widely used by the target populations. The majority of key informants (12 of the 18 that provided a numeric ranking) indicated that the program has increased the use of SMT products by end-users to a significant degree. For example, key informants noted the following:

  • An increased availability of satellite data has allowed the SMTSA to broaden and customize the forecast products it can supply to users, and allowed private industry providers to develop products for targeted populations (e.g., oil rigs), though there is not yet sufficient evidence to conclude whether this has led to increased usage among targeted populations.
  • There is increasing interest in forecast information and products at marine boating shows.
  • Many TV stations serving maritime populations now read weather warnings.

As noted in Figure 1, evidence from the survey of end-users also suggested that SMT SA products are used by targeted stakeholders. The ECWeatheroffice website, radio broadcasts, Weatheradio, and television were frequently identified as SMTSA products used by end-users, and the EC Weatheroffice website was identified as the main source of weather information by a large majority.

As noted in Figure 2, respondents to the survey of end-users also provided high ratings (using a five–point scale) on the accessibility, usefulness, ease of understanding, timeliness and accuracy of the weather information that they used. Respondents generally provided a rating of either 4 or 5 for each item. Usefulness and ease of understanding were most likely to be rated 5 (excellent), by 53% and 50% of respondents, respectively. Respondents provided marginally lower ratings on accuracy than the other items, as shown below, with only 6% rating this item as a 5, although nearly 56% provided a rating of 4.

Figure 2: Rating of the Marine Weather Service (text description)

Figure 2: Rating of the Marine Weather Service

Other sources of evidence, such as the 2004 Marine Weather National Services Survey, also suggest that historically the use of SMTSA products and services has been high. Some findings from the survey:

  • Of the 98% of mariners that use weather forecasts, 98% use marine weather forecasts an average of three times per day.
  • The daily ice charts (86%) and recommended ice route (75%) are important ice information products (rating of four or five on a five–point scale). The thirty-day forecast (35%) and seasonal outlook (34%) are rated as less important.
  • Half of participants changed their plans in response to marine weather information "once in a while," one third "most of the time," and one tenth "almostalways."
  • Two thirds (66%) of participants either "always" (29%) or "most of the time" (37%) followed the recommended ice route when operating in the ice.

Outcome 3: Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice

An unqualified assessment of this outcome was not possible, given the difficulty in establishing direct attribution of emergency interventions to SMT programs and activities. Nevertheless, the evaluation team was able to examine evidence from marine investigations of the Transportation Safety Board and key informant interviews that referred to general trends linking weather and ice information to marine safety. From 2005 to 2009, there were 36 marine investigations published by the Transportation Safety Board. Of these reports, most incidents (75%) tended to occur during favourable weather conditions. Of those involving inclement weather or difficult seas, weather reports were accurate and had given sufficient notice to mariners in all but one case, where an inaccurate report was deemed to have increased the risk of an accident.

Table 10: Transportation Safety Board–Summary Table 2005-2009 (text description)
YearTotal no. reportsNo. incidents where weather/ice conditions were a factorNo. reports where service provided by MSC was accurateSummary of incident
2009711Quartering seas
200870NA 
2007511Quartering seas
2006124380Squall
Quartering seas
Hurricane-force winds (as predicted by U.S. Coast Guard)
Strong winds81
20051533Fog
Caught in ice
Poor weather
Total4698 

Key informants were generally cautious in addressing this outcome indicator directly, and several respondents did not provide a rating on the degree to which the SMT SA contributed to reducing emergency interventions, noting that the role played by EC is important, but by no means the only factor in reducing the number of emergency interventions.

Of those who provided a view regarding the degree, responses ranged from moderate to significant. A number of examples were provided of how SMT products and services have led to reductions in the number of emergency interventions associated with marine weather or ice, including:

  • lobster season no longer has a pre-set opening day--it varies depending on the forecast, to protect the safety of mariners;
  • emergency intervention representatives consult EC much more frequently than previously and make greater use of EC's information for planning; and
  • ice information was used to issue a warning and help in a situation regarding a missing teenager in the north, by enabling the search areas to be better targeted.

Also, all three key informants from OGDs were generally positive regarding the MSC's role in providing accurate weather information to assist and support maritime emergency responses.

Outcome 4: Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice

The majority of respondents do not believe that this outcome can be assessed at this point in time, because there are no data available to address the indicator, and it would be difficult to know how weather and ice information could have potentially affected the advanced planning for travel routes. Examples of reductions in unexpected delays that were nonetheless noted by respondents included: revisions to ferry service schedules in the Pacific and Atlantic regions as a result of forecasted information, and a reduced need for ship navigation through ice-bound waters as a result of regional CCGadjustments to shipping routes; these adjustments were based on detailed information provided by the SMTSA. Key informants noted, however, that the MSC plays a limited role in affecting this area of maritime navigation and that, at the very least, this outcome needs to be considered a shared outcome with other parties.

CCG documentation indicates that only a small minority of service delays for clients in 2008-2009 were due to ice conditions (13%) or weather or tide conditions (4%). Once again, though, this information is not adequate to determine whether these delays were anticipated or unexpected, or what role SMT-related products may have had in mitigating their impact.82

Unintended/unanticipated outcomes

Only one unintended outcome of the SMT SA was identified by multiple key informants. This issue related to unforeseen impacts on service delivery as a result of introducing a standardized national approach to the forecast for the Weatheradio broadcasts. Prior to the initiative, which introduced a standardized two–day forecast on all broadcasts, there had been a three-day forecast included in broadcasts in the Atlantic region, and a one–day forecast on broadcasts in other parts of the country. The modification to this broadcast schedule in the Atlantic region and the resulting frustration among clientele in the region had been unintended.

External factors affecting the achievement of outcomes

A number of issues were identified by key informants as areas outside of the MSC's direct control that have, or may have, impacts on the performance of the SMTSA. Approximately one quarter of the key informants pointed to the risks of relying on other collaborating organizations for funding and service delivery. For example, it was noted that EC is increasingly relying on external organizations for the provision of satellite monitoring data, which have become an important source of meteorological information. Furthermore, SMT SA services could be greatly reduced, at least in the short term, if an existing collaborating organization (CCG being the most frequently cited) were to undergo budget cuts or a change in business model. Another cited example was the fact that EC relies on CCG for transportation when servicing weather buoys, and there have been instances where this has resulted in several months passing without service for a particular buoy, thereby resulting in a loss of monitoring data.

4.2.3 Demonstration of Economy

Demonstration of Economy
Evaluation Issue: PerformanceIndicator(s)MethodsRating
6. Are Services to Marine Transportation achieving their intended outcomes in the most economical manner?
  • Views on whether the resources and inputs into the program are managed economically
  • Clearly defined and understood governance structure, including program processes, roles, responsibilities and accountabilities
  • Views on whether roles, responsibilities and accountabilities are functioning as intended
  • Views on the appropriateness of program activities, processes, and governance structures
  • Document review
  • Financial analysis
  • Literature review / international comparison
  • Key informant interviews
Progress Made; Attention Needed

Generally, the SMT SA appears to be managing its resource inputs in an economical way, including reallocating resources from existing areas to address new initiatives, as was the case with METAREAs Initiative funding for 2010-11. Additionally, effective governance and priority-setting mechanisms have been established. There are, however, some areas where further clarification of roles and responsibilities may be beneficial, particularly regarding certain responsibilities between ECand DFOrelated to marine weather issues for buoy maintenance and storm surges.

Resource Allocation

The numbers in Table 11 represent expenditures for the SMT SA for the last four fiscal years. The budgets at the start of each fiscal year were not available from the Department, thus precluding an analysis of budgeted versus actual expenditures; however, the Evaluation Division was informed by the Financial Services Division that this SA tends to manage its budget very closely and that actual expenditures essentially equate to the total budget for the program. As noted previously, the METAREAs Initiative is outside the scope of the evaluation as it is too new to evaluate, but expenditures for METAREAshave been included in Table 11, in order to provide a complete picture for the 2.2.2 PAA element (the SMT SA).

Table 11: 2007-2009 Annual Financial Expenditures for 2.2.2 Services to Marine Transportation83 (text description)
Financial Expenditure2007-082008-09
MarineIceTotalMarineIceTotal
FTEs319512632108140
Salary2,3862,8615,2472,6392,7715,410
O&M1842,1192,3022122,6342,846
Capital077770337337
VNR-Salary13,8073,80805,2005,200
VNR-O&M05,7165,71605,1195,119
TOTAL*2,57014,57917,1502,85116,06218,912

 

Table 11b: 2009-2011 Annual Financial Expenditures for 2.2.2 Services to Marine Transportation83 (text description)
Financial Expenditure2009-102010-11
MarineIceTotalMarineIceMet-
areas
Total
FTEs32108140321047144
Salary2,8563,0375,8933,0932,5886566,336
O&M1241,5141,6381141,8692352,218
Capital01161160355177532
VNR-Salary55,5885,59405,26105,261
VNR-O&M05,1165,11624,13824,142
TOTAL*2,98515,37218,3573,20914,2121,06918,491

* Numbers are shown in $000s (except FTEs).

Of note, approximately two thirds of the budget for ice services comprises VNR funding, which primarily represents fees for services to CCG. Although the partnership with CCGbrings many benefits, this reliance does present an ongoing risk to the SMTSA, as its ability to deliver on mandated services is highly dependent on this continued funding. Furthermore, it poses a further challenge for long-term expenditure planning, as SA program management must ensure flexibility to adapt to possible changes in funding levels due to CCGbudget variations. Additionally, the SMTSA, and especially the marine weather component, is reliant on the MSC's Weather, Observations, Forecasts and Warnings SA (PAA element 2.1.1), as work conducted by meteorologists and other MSC specialists to support services to marine transportation is built upon the "core" services resourced under this SA. The SMT SA could not perform its work at current resource levels without leveraging the work done by this other SA.

Also of note, although FTEs did increase moderately in 2010–11 with the introduction of the new METAREAs component of the program (additional 4 FTEs, representing less than a 3% increase), it did not come with an overall budget increase for the PAA element. Rather, expenditures for the Ice Research and Ice Warnings and Forecast Information components of the program were reduced by essentially the same amount as the funding now identified for the METAREAs Initiative. Given that the METAREAs Initiative is in early implementation, it is too early to assess its impact on the achievement of results.

As demonstrated by the numbers in Table 11, and portrayed in Table 12, the SA had an increase in expenditures of approximately 10% between fiscal years 2007–08 and 2008–09, for both the marine and ice components of the SA. For the remaining two years of the study period, expenditures for the SAremained relatively stable year over year, although there was a slight fluctuation between the two components.

Table 12: Year over Year Changes in Expenditures(text description)
Expenditure2008–09 vs. 2007–082009–10 vs. 2008–092010–11 vs.
2009–10
MarineIceTotalMarineIceTotalMarineIce**Total
Year over year changes in spending11%10%10%5%-4%-3%8%-1%1%

 ** Includes Ice Research, Ice Warnings, Forecasts & Information and METAREAs.

Table 13 indicates the percentage distribution of resources across the three branches that receive financial resources under the SMT SA. There has been little change in the distribution of financial resources, either to the program components or across the three branches, since 2007–08. There were no comments from key informants suggesting that this distribution was likely to change, barring concerns regarding a loss of support from the key partner, i.e., the CCG. The low degree of variation across the four years appears to suggest that there does not appear to be a need to adjust the SMT SA's overall operational approach.

Table 13: Percentage distribution of resources over last four fiscal years84 (text description)
Distribution2007-082008-092009-102010-11
MSCS&TCIOBMSCS&TCIOBMSCS&TCIOBMSCS&TCIOB
Marine15%0%0%15%0%0%16%0%0%17%0%0%
Ice69%1%15%69%1%16%68%1%15%66%1%16%
% of Yearly Total84%1%15%84%1%16%84%1%15%84%1%16%
Governance and Priority Setting

Though generally positive, there were a variety of responses regarding the extent to which the SMT SA governance and priority–setting mechanisms were supporting the effective management of resources. Since June 2010, the main governance mechanism for the SA has been the Marine Weather and Ice Program Management Board. About one quarter of the key informants were positive in their appraisal of the SMT SA's governance and priority-setting structure, noting that the new board has been effective in setting priorities, such as approaches to issuing marine forecasts and watches. The remaining key informants indicated that the newness of the board limited the extent to which it was either willing or able to comment on its functioning, although there was a general sense of optimism that the new board structure would be effective in managing SMTSA programs and services.

EC key informants were also asked to comment on the extent to which the two SMT SA components (weather and ice) could share resources and expertise in order to minimize costs and increase effectiveness. It was generally noted that the training required for a weather meteorologist is significantly different than for an ice specialist and, aside from co-locating the two functions, other efficiencies, such as sharing or transferring staff from one area to another, would be impractical.

Several key informants noted, however, that there is some duplication in terms of weather analysis as it relates to wind forecasts. Ice specialists use wind speed direction to track ice flow direction and, as the CIS uses remote sensing to track ice and wind, there are opportunities for both the weather and ice components to share these data. Others also noted that there is room for more integration, as there are natural reasons why ice and marine weather information should be more intertwined (for example, ice changes shoreline impacts and storm surges). Further, end-users do not separate the two functions, so opportunities may exist for additional coordination of information dissemination, warnings, websites, modelling and electronic charting.

It was noted, however, that there are expectations that the METAREAs Initiative will contribute to finding efficiencies in terms of how the marine forecast and ice components work together, and to finding opportunities to work with EC's S&T community toward developing further services unique to northern communities. Given the early implementation of this initiative, however, it was not possible to discuss examples of how the initiative has yielded these efficiencies.

Clarity of Roles and Responsibilities

The clarity of roles and responsibilities among federal government stakeholders involved in the delivery of services to marine transportation was assessed. Key informants generally pointed to a clear articulation of roles and responsibilities. Several respondents noted, however, that there is room for improvement in this area, specifically the need for clarity regarding the formality of shared responsibilities and costs between EC and its collaborators. The federal government's ice services program is delivered through a formal MOU between EC (specifically the MSC) and DFO (specifically the CCG),85 which generally helps define roles. It was noted by EC and DFO respondents that the CIS relationship with the CCG is generally working very well: both parties understand what each is expected to deliver, due to the existing 5–year MOU between the two parties.

However, key informants noted that there were still too many informal arrangements in place between SMT and its collaborators, and identified a lack of clarity in some areas, particularly for the marine weather component where the definition of roles and relationships is much more informal. The most frequently identified areas in need of increased clarity were regarding responsibilities for deployment and maintenance of weather buoys and storm surge warnings:

  • Deploying and servicing buoys – Key informants noted that the CCG has discussed charging EC to deploy weather buoys. Key informants at both EC and DFO used the term "ahandshake agreement" to define the current relationship between the two departments regarding the displacement of buoys. Although this relationship has historically been effective, if resources become increasingly stretched this informal agreement may no longer be feasible and EC could be faced with an additional, unexpected cost to deliver on SMT services. There is currently no national MOU for servicing buoys, though one does exist for Pacific and Yukon Region, which key informants noted was primarily developed as a result of existing relationships at the managerial level. On a related note, key informants also noted that there is some confusion as to who is responsible for the database of buoy information for record keeping and historical analysis.
  • Storm surge – The storm surge warnings issue arises from the fact that delivery of this capability is not consistent across all regions and is not clearly defined. Currently, EC provides the publicly accessible data86 and warnings to the east coast and Great Lakes, while DFO provides this information to the west coast and Quebec. A key difference noted by key informants in relation to this responsibility was that the MSC is an operational organization maintaining an operational staff 24/7, whereas DFOactivities tend to be conducted by scientists solely during regular business hours.

Another area where clarity could be improved, to ensure that program costs are minimized, is contingency plans for emergency situations (for example, oil spills); these plans are not as clear as many (including external stakeholders) would like. Given the potential costs associated with a lack of clarity over emergencies, there were calls to have roles clearly laid out between all key players (e.g., EC, DFO, TC and DND) when an incident occurs in Canadian waters. A need was also identified for better clarity over responsibilities in the Arctic, particularly as the costs associated with working in the North are higher than in southern Canada. A noted example was the need to find the most cost-effective solutions for deploying weather buoys and maintaining a minimum level of verification through observations, although, again, it was identified that the METAREAs Initiative will likely address this.

4.2.4 Demonstration of Efficiency

Demonstration of Efficiency
Evaluation Issue: PerformanceIndicator(s)MethodsRating

7. Is the Services to Marine Transportation Program undertaking activities and delivering products in the most efficient manner?

  • How could the efficiency of the program's activities be improved?
  • Are there alternative, more efficient ways of delivering the program?
  • Views on whether good value is being obtained with respect to the use of public funds
  • Views on how the efficiency of program activities could be improved
  • Views on whether there are alternative, more efficient ways of delivering program activities and outputs
  • Document review
  • Financial analysis
  • Literature review / international comparison
  • Key informant interviews
Achieved

The evidence suggests that the program is generally delivering its services in an appropriate and efficient manner and has taken steps to improve efficiency. There may be opportunities for further efficiencies associated with new technologies for dissemination, although this must be carefully managed so that mariners who rely on older technologies continue to receive the information they require for safe and efficient operation.

Overall Views of Efficiency of SMT SA Products and Services

Generally, key informants expressed a positive view regarding the efficiency of the SMT SA, and noted several examples:

  • The majority of key informants, including program representatives and end–users, expressed views on the value of SMTSA products and services with respect to the use of public funds, by noting that the quality and production of forecasts and warnings are high and provided in a format and language that is understood by the marine community, and that the delivery of services is appropriate and efficient given available resources and the need to provide services 24/7. This is consistent with findings from a recent evaluation which found that EC's weather prediction service is generally viewed as providing high-quality information and products.87
  • It was noted by EC and CCG key informants that the CIS has been receiving decreasing funding from the CCG to maintain the same level of service (although funding was stable for the four–year period of the evaluation, there has been a decrease of approximately $7 million since the MOU in 1995). Evidence from the key informant interviews and document review suggests that the main reason that the CIS has maintained the same level of service despite budget cuts has been cost savings associated with the increased use of radar and remote sensing technology and the decrease in the use of fixed-wing aircraft observation.
  • Two key informants also noted that international collaboration has been effective, particularly in the Great Lakes region, although more could be done with the United States in developing common products and dissemination mechanisms. No international stakeholder was interviewed for this evaluation, although a senior official in the U.S. federal government was interviewed recently in a related evaluation,88 commenting that CIS ice observation products have improved dramatically, primarily because they have much better sources of information now that they use radar and remote sensing technology that has revolutionized the entire field. Furthermore, this official noted that Canada and its U.S. counterparts have solidified their relationship and that the organizations are undertaking collaborative projects and expanding joint activities.
  • Dissemination systems were noted as working reasonably well given the variety of dissemination tools offered. Two key informants expressed the view that the dissemination methods for the marine forecasts were meeting needs for this targeted group of users better than the dissemination methods available for broader public offering.
  • Consultations with other government agencies (e.g., Canadian Marine Advisory Council) and communication at trade shows have been an effective, low-cost approach to improving the reach of SMTSA products and services, and to providing staff with the opportunity to engage with the maritime community.

Key informants also identified several areas that can potentially improve efficiency:

  • The need for a broad range of dissemination systems was the most frequently noted. It was noted that the SMT SA is obliged to deliver its information in a wide variety of systems to meet the diverse needs and technological capacities of its end–users. For example, radio is still considered a key source of dissemination for mariners well out of port, though there are limitations due to expense and reach. Meanwhile, the Weatheroffice website was the most highly used source of marine weather information among external users surveyed, and several program representatives believed there were opportunities to explore more technologically advanced capabilities such as opportunities to leverage mobile phones. Although no solution was provided, many believed that improvements to the dissemination system would improve the efficiency of SMT SA services, through leveraging new technologies.
  • Equally cited by key informants was the topic of public-private partnerships. A growing number of firms obtain publicly accessible data from EC sources to produce very specific and/or localized forecast information for specific clientele, such as providing information on the safety limits of the ice edge to remote northern communities (the evaluation team found six examples of such organizations and spoke to representatives of two such groups). EC and industry representatives commented during the open-ended interviews that the provision of specialized information would push EC's existing resources too far, and that private industry was in a better position to meet the demands of specific clientele. Although one key informant suggested that EC could also raise revenue by charging a fee-for-service to deliver similar localized information, the majority of respondents who commented on this issue suggested that the current approach, in which EC manages the general network of national data and private providers use this data to address specialized needs in the market, appears to be effective and is more in line with open government policies in Canada and the United States.
  • Five key informants indicated that the SMT SA needs to find approaches to improve verification and observation networks because data availability is sparse, particularly in the North.

Key informants did offer several alternative approaches, however, to improve efficiency and maximize the use of current resources. The majority of alternatives were based on expansion or better use of technology, which is perhaps not surprising given the cost savings and improved efficiency for SMT SA services based on the increased use of technological innovations such as satellite-based data.

Approximately one quarter of the key informants indicated that the SMTSA should explore the better use of technology for dissemination, such as applications for smart phones or alerts via text messages to cellphones. Though there will be technical challenges to making use of enhanced dissemination products, key informants indicated that there is a need to look at the availability and feasibility of new technologies and that both SMTSA components need to take better advantage of technological advances and innovations. However, as was noted above, given the wide range of end-users, many mariners still use and are attached to older sources of technology, making it difficult and potentially harmful for the SMT SA to replace one dissemination tool with another.

Comparison of Costs

As a final source of evidence related to the efficiency of SMTSA products and services, the evaluation team examined an external benchmarking study of the meteorological service organizations in seven countries, including Canada's MSC. Although the study is not specific to marine and ice information, it contained an international cost comparison for the provision of meteorological services among the organizations shown in Figure 3.89 The figure shows relative spending, not only in terms of the cost per capita, but also the cost per square kilometre.

Figure 3: Relative Spending on Meteorological Services90 (text description)

Figure 3: Relative Spending on Meteorological Services

The United States and Finland are of particular interest to the SMTevaluation, as those countries' meteorological service organizations also have ice services. Regarding Finland, its Finnish­ Meteorological Institute (FMI) experienced budget increases from 2007–2009, mainly due to increases in external funding and extra government revenue to increase services to the marine sector and research for the public good. The FMI provides many of the same services to marine transportation as EC's MSC, including providing forecasting models to its coast guard, among other clients.91Further, the FMI website contains information on marine weather and the Baltic Sea, including weather forecasts for shipping, sea level, ice conditions, wave forecasts, and sea temperature forecasts.92

Figure 3 indicates that Canada spends approximately twice as much per person as does the United States and approximately half as much as does Finland. In terms of spending per square km, Canada spends less than both the United States and Finland.


21. EC and DFO. 2005. IISPA.

22. CCG. 2009. CanadianCoast Guard Fleet Annual Report – 2007-2008, p. 12.

23. These issues are discussed in detail in section 4.1.3.

24. United Nations. 1982. United Nations Convention on the Law of the Sea.

25. IMO, 1974 and later amendments. International Convention for the Safety of Life at Sea.

26. Article 4 and 5 of the Department of the Environment Act, as quoted in WES Consultation Document August 28 2008.

27. WES Consultation Document August 28 2008.

28. Standing Senate Committee on Fisheries and Oceans. 2009. Rising to the Arctic Challenge: Report on the Canadian Coast Guard, pp. 21-27, 42-45.

29. Government of Canada. 2010. The Budget Speech 2010: Leading the Way on Jobs and Growth.

30. Government of Canada. 2010. Speech from the Throne. Delivered March 3, 2010.

31. Government of Canada. 2007. Speech from the Throne. Delivered January 26, 2009.

32. Finance Canada. March 4, 2010. The Budget Speech 2010: Leading the Way on Jobs and Growth, pp. 106, 68, 111, 255, 231.

33. Terms of Reference - MIPMB v 20100608.

34. As noted previously, the SMT PAA element is the entity being focused on in this evaluation.

35. EC. 2006. Environment Canada's Sustainable Development Strategy 2007-2009, p. 4.

36. Office of the Auditor General of Canada. 2008. 2008 December Report of the Commissioner of the Environment and Sustainable Development, Chapter 2--Managing Severe Weather Warnings--Environment Canada.

37. EC and DFO. 2005. IISPA, p. 5.

38. In 2008, EC obtained certification for its weather and environmental services function under ISO 9001 following implementation of its QMS.

39. EC. 2010. 2010-2011 Performance Measurement Framework Weather and Environmental Services.

40. EC and DFO. 2005. IISPA. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

41. EC. 2008. Weather and Environmental Services QMS Manual.

42. EC. 2010. Meteorological Service of Canada: Marine Forecast Verification Project – Project Charter.

43. A draft PMF was under development during the evaluation and was shared with EC evaluation team members.

44. EC. 2010-11. Performance Measurement Framework – Weather and Environmental Services.

45. The 2010-11 draft PMF outlined the following outputs and outcomes: percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines; lead times of marine gale warnings of 24 hours or greater, in 80% of events; and level of satisfaction of the marine communities with respect to weather, wave and ice information received by EC. Other outputs and outcomes were developed from PAA element descriptions, evaluation scoping interviews, and consultation with the Departmental Evaluation Committee.

46. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

47. Earlier wording in reports: "the number of duty days delivered versus requested," which is consistent with wording in IISPA: "# duty days delivered vs. requested (%)."

48. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

49. EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

50. Defined as the number of correct warnings as a percentage of the total number of events.

51. Defined as the percentage of warnings that are incorrect.

52. EC. Maritimes Verification and Newfoundland Verification, 2007-2010.

53. EC. 2004. Marine Weather Services National Survey. Final Report.

54. Recreational mariners represent a minority of the total mariner population and are not the primary focus of the SMTmandate.

55. DFO. 2009. Icebreaking Program Report on Performance: Winter Operations 2008-2009.

56. EC. 2011. Weatheroffice website statistics.

57. Transportation Safety Board. 2011. "Marine Statistics."

58. DFO. 2009. Icebreaking Program Report on Performance: Winter Operations 2008-2009.

59. EC. 2004. Marine Weather Services National Survey. Final Report.

60. DFO. 2008. Levels of Service Review: Summary of Client Comments. Draft Report.

61 EC. 2010. Canadian Marine Advisory Council Maritimes Region Marine Day.

62. EC. 2011. North Atlantic Fish & Workboat Show 2010 November 26-27, 2010.

63. DFO. April 2011. Evaluation of the Icebreaking Service.

64. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

65. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

66. EC. 2004. Marine Weather Services National Survey. Final Report.

67 EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

68. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

69. EC. CIS Monthly Performance Measurement Reports, July 2007 to Aug 2010.

70. EC. 2010. Marine and Ice Program Management Minutes Sept. 24 2010.

71. EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

72. EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

73. EC. 2004. Marine Weather Services National Survey. Final Report.

74. EC. 2011. Weatheroffice website statistics.

75. EC. 2004. Marine Weather Services National Survey. Final Report.

76. EC. 2010. Canadian Marine Advisory Council Maritimes Region Marine Day.

77. EC. 2010. Canadian Marine Advisory Council Maritimes Region Marine Day.

78. EC. 2011. North Atlantic Fish & Workboat Show 2010 November 26-27, 2010.

79. EC. 2011. North Atlantic Fish & Workboat Show 2010 November 26-27, 2010.

80. Includes one report where data were provided by the U.S. Coast Guard.

81. As noted in the description, unexpected strong winds contributed to the single maritime accident in which weather information was deemed to be inaccurate.

82. DFO. 2009. Icebreaking Program Report on Performance: Winter Operations 2008-2009.

83. The figures shown as "Marine" represent all line items that map to Marine Weather Warnings Forecasts and Information in the departmental financial coding system, and the figures shown as "Ice" represent the values for both Ice Research and for Ice Warnings, Forecasts and Information.

84. Source: Financial Management Branch, EC. May 2011.

85. EC and DFO. 2005. IISPA.

86. Storm Surge

87. EC. 2011. Evaluation of Support of Weather Prediction.

88. EC. 2011. Evaluation of Support of Weather Prediction.

89. Kerry A., Wilkinson C., van Mossel J. June 4, 2009. Benchmarking Study on Business Models for the MSC. Final Report. p. 21.

90. Ibid., p. 21. Note: amounts are presented in U.S. dollars.

91. Ibid., p. 9.

92. Wind and pressure forecast for Baltic Sea

Return to Table of Contents

Conclusions

Relevance

Multiple sources indicated that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions.

Findings from multiple sources indicated that the SMT SA is aligned with the legislated mandates of the involved departments, and with responsibilities assigned through agreements and other mechanisms for collaboration. Furthermore, the evidence indicates that the MSC is obligated by various acts and agreements to deliver SMT SA services, including in the context of ongoing northern development.

For the most part, evidence found that EC’s SMT SA services complement services in other government departments. However, concerns were expressed by EC and collaborating department representatives regarding the extent to which some areas of shared responsibility are clearly articulated. There is additional concern that the Department’s ability to deliver on its responsibilities is vulnerable, as these services are reliant on key partners or collaborators.

In addition, the SMT SA is aligned with and supports federal government priorities related to health and safety, the economy, infrastructure and sovereignty, as well as departmental strategic outcomes and priorities related to ensuring that Canadians are equipped to make informed decisions on changing weather, water and climate conditions.

Key informants noted that one area of expanding interest for the federal government, and an area in which the MSC will likely provide increased marine weather and ice services, is northern development. The MSC has been resourced to undertake new approaches to providing services to Arctic communities and industries under a newly established METAREAs Initiative that is funded under this SA. However, as this initiative had only begun at the outset of the evaluation, it was not included in the analysis and will be evaluated by EC at a later date.

Performance

The evaluation found evidence of a strong commitment to performance measurement, supported by the MSC’s ISO 9001 Quality Management System. Considerable data collection is occurring, and program management appears to recognize the value of performance measurement.

Although the activities that comprise the SMT have existed within the MSC since its origins 140 years ago, the grouping of the provision of marine weather information and ice information as one PAA element, known as the SMT, was just recently introduced with the 2010-11 PAA. As such, the existence of a Performance Measurement Strategy that presents a logic model for the PAA element and a cohesive PMF for the entire element does not yet exist, although efforts are being made in this area.

Nevertheless, the evaluation found that the SMT SA was able to provide data and targets on key outputs that were identified for the purpose of the evaluation, such as the delivery of marine weather warnings and ice forecasts and charts, and that, furthermore, it is performing at or near the vast majority of these targets. With regards to the availability of performance measurement related to outcomes, data were also available for the outcomes identified for the purpose of the evaluation, although baseline data that would support an understanding of how performance has changed over time was limited.

The evaluation found evidence of strong performance regarding outcomes related to usefulness and ease of understanding of weather and ice information products. There is a very high incidence of mariners accessing and making use of the marine weather and ice information provided by EC, and overall satisfaction with the products and services was high. Marine and ice information is disseminated to mariners with a broad range of methodologies. Although the EC Weatheroffice website was identified as the most frequently used source of marine weather information, evidence found that most mariners relied on more than one source or technology for their information needs, and further, that users of marine information are a diverse group who include both those who rely on receiving information in the same manner in which it has been provided for many years, as well as those who would like to see further enhancements that leverage advanced technological capabilities. This presents a challenge for EC from an efficiency perspective, as it makes it very difficult to find cost savings to fund new advancements from the removal of historic products.

The SMT SA operates on a budget of approximately $18.5 million per year, and spending on these activities has been relatively stable over the four years of the study period. EC’s SMT is not, however, a self-contained function, but rather is able to deliver on its outcomes by relying both on the broader underlying capacity of the MSC for weather and environmental services as well as its important relationships with its key partner, the CCG, and many other collaborators and stakeholders, including TC, the CSA, the WMO and the U.S. National Ice Center, among others. These relationships and the cooperative, collaborative manner in which the Department operates in this area, supports EC’s ability to deliver marine transportation services in a cost-effective manner. The relationship with the CCG appears strong, for the most part, although there are areas where roles and responsibilities could be clarified. In particular, clarifying roles related to servicing weather buoys and providing storm surge warnings were identified as areas lacking clarity. Addressing areas of ambiguity will have the added advantage of ensuring that the inherent risk associated with reliance on another party is minimized.

Return to Table of Contents

Recommendations

The following recommendations are based on the findings and conclusions of the evaluation. The evaluation recommendations are directed to the ADM, MSC, in view of the ADM’s responsibility for the overall management of SMT activities.

Recommendation 1

During the conduct of this evaluation, the evaluation team was able to rely on a variety of performance data from the SMT SA’s initial draft 2010-11 PMF to develop a preliminary assessment of various aspects of the SMT SA’s performance. Working with this, the evaluation team developed a set of outputs and outcomes in order to support a preliminary assessment of performance. As noted in the report, there was some concern expressed by SMT management at the onset of the evaluation that the outputs and outcomes used for the evaluation may not be adequately aligned with SMT services and activities, and that management may not be able to collect the required data. It is notable that the program was able to produce data across these outputs and outcomes, and this does represent a significant step toward ongoing performance reporting. Additionally, it is clear that the SA recognizes the importance of formalized performance measurement, through ongoing development of the SMT SA’s PMF. However, several gaps in performance reporting still remain, including a more thoroughly validated set of performance indicators across expected outputs and outcomes, and a reporting strategy that integrates these data sources into a common reporting tool. It is therefore recommended that the ADM MSC, using the evidence presented in this report as a baseline for performance reporting, continue to develop the PMF, including a logic model, expected program outputs and outcomes, and a strategy to collect and report on this information. Attention should be given to the performance reporting requirements of organizations such as the CCG.

Recommendation 2

The services provided by SMT are mandated by a range of obligations set out in both international and domestic acts and agreements. Furthermore, these acts and agreements link SMT to a variety of federal and international collaborating organizations, and have clearly formalized several key areas of SMT service provision (such as the deployment of ice specialists onboard CCG ships). Although obligations clearly establish the need for SMT to provide maritime weather and ice services, there are nevertheless potential risks to SMT if there are changes in the level or type of service provided, or in the financial and human resource commitments by key collaborating departments. During this evaluation, there was particular concern expressed regarding clarifying the relationship with DFO/CCG in areas such as the maintenance and servicing of buoys, and the provision of storm surge warnings in various regions. It is therefore recommended that the ADM MSC develop a strategy to identify and address risks posed to the SMT SA due to any areas of uncertainty in its relationships with collaborating departments. As a starting point, this strategy should include an approach to formalize the relationship with DFO/CCG regarding servicing weather buoys and providing storm surge warnings.

Recommendation 3

It was frequently noted that the SMT SA must use an increasingly wide range of communications technologies to disseminate meteorological information across all potential users, and that program management is limited in the number of media that can be reduced or removed, as it may pose safety risks to end-users that rely on these specific forms of media. Given the growth in available potential methods of dissemination in relation to current resources, it is therefore recommended that the ADM MSC establish a strategy to manage the evolution of dissemination strategies for marine weather and ice information, and to examine the consequences of reducing or removing current dissemination mechanisms. This strategy should be based, to the extent possible, on the documented needs of SMT’s maritime clients and should include cost estimates for each proposed new approach. A process should be established to ensure that the strategy is reviewed and revised on a regular basis.

Return to Table of Contents

Management Response

Recommendation 1

It is recommended that the ADM MSC, using the evidence presented in this report as a baseline for performance reporting, continue to develop the PMF, including a logic model, expected program outputs and outcomes, and a strategy to collect and report on this information. Attention should be given to the performance reporting requirements of organizations such as the CCG.

Attestation of Agreement/Disagreement with the Recommendation


The MSC agrees with the recommendation.


Management Action


The MSC already has a PMF in place for the SMT program (PAA sub-activity 2.2.2), and has been updating the program outputs and outcomes as well as strategies to collect and report on this information, for several years. Updates to the expected outcomes and outputs for the 2012-13 reporting cycle are awaiting EC management approval, and are being submitted to Treasury Board Secretariat in the summer of 2011.

As part of the development of a performance strategy for the METAREAs marine services (a new initiative that falls under the SMT program), a logic model and PMF were developed during 2010-11. Through this exercise for the METAREAs, there is transferable information that will be used in the development of a logic model for the SMT program. The MSC is committed to delivering on a logic model and updating its Performance Measurement Strategy by March 2013, after sufficient consultation with its clients and partners.

In the initial IISPA between the MSC’s Canadian Ice Service and the CCG, a logic model and performance measures were developed. Reporting on all of them became challenging for both, and thus, in the re-negotiated IISPA, the number and type of performance measures have been reduced and focused significantly, with the intention to refine them in the coming year.

Pre- and post-season meetings with the CCG and clients result in requests for new products, and reductions and/or changes in other products. These are usually captured in the minutes of these meetings and actioned by either party.


Timeline
Deliverable(s)
Responsible Party
March 2012
Logic model for SMT program finalized
DG, WEPS
March 2013
Updated PMF outputs, outcomes, and performance data collection and reporting strategy finalized
DG, Business Policy (or DG, WEPS)

Recommendation 2

It is recommended that the ADM MSC develop a strategy to identify and address risks posed to the SMT SA due to any areas of uncertainty in its relationships with collaborating departments. As a starting point, this strategy should include an approach to formalize the relationship with DFO/CCG regarding servicing weather buoys and providing storm surge warnings.

Attestation of Agreement/Disagreement with the Recommendation


The MSC agrees with the recommendation.


Management Action


The MSC enjoys benefits and synergies from its collaborative arrangements, but also acknowledges the risk posed by uncertainties with respect to its relationships with some collaborating parties, including other government departments. This risk has been identified in the WES Risk Management Profile, although the risk statement does not specifically note the weather buoy and storm surge warning elements associated with the relationship with DFO/CCG. This dimension of the collaboration risk will be more fully reflected in the next update to the Risk Management Profile. The MSC’s Marine and Ice Program Management Board will be tasked with developing strategies to analyze and mitigate these risks, and to present a strategy to MSC senior management before the end of 2011-12.


Timeline
Deliverable(s)
Responsible Party
December 2011
Reflect collaboration risks of SMT program in the WES 2012-13 Risk Management Profile
DG, Business Policy
March 2012
Analysis and strategy document dealing with collaboration risks in the SMT presented to MSC management committee
DG, WEPS

Recommendation 3

It is recommended that the ADM MSC establish a strategy to manage the evolution of dissemination strategies for marine weather and ice information, and to examine the consequences of reducing or removing current dissemination mechanisms. This strategy should be based, to the extent possible, on the documented needs of SMT’s maritime clients and should include cost estimates for each proposed new approach. A process should be established to ensure that the strategy is reviewed and revised on a regular basis.

Attestation of Agreement/Disagreement with the Recommendation


The MSC agrees with the recommendation.


Management Action


There is a risk, when in the business of information dissemination, of chasing the newest technologies (e.g., social networking) without fully understanding client needs and the effectiveness of the technologies in meeting these needs. The MSC’s dissemination strategies take into consideration a number of factors, including client needs, technological opportunities, synergies among service areas, partnering opportunities (e.g., with CCG Marine Radio) and fiscal realities. The needs of mariners have had a major influence on our dissemination systems--most notably the design of the Weatheradio network, which has a significant mariner audience. We recognize that the marine community has unique requirements (e.g., often not having access to cell phones and the Internet when at sea), hence some solutions for public weather information dissemination are inadequate for the marine and ice program.

Strategies for dissemination should be based on broader service strategies, and the MSC is in the process of documenting a new service strategy. Once approved, this strategy will provide guidance with respect to many service directions, including dissemination strategies. In the meantime, MSC dissemination experts continue to work with the telecommunications industry to stay abreast of the latest technological advances, while client relations personnel, including those serving the SMT program clients, interact with client organizations to ascertain how their information–access needs are changing. One example of an outcome resulting through these processes is the redesign and reconfiguration of the Weatheradio network in Atlantic Canada, which is currently under way. These changes will see more region-specific programming from each transmitter location, hence reducing the program cycle (the length of which has been a frequent complaint from mariners).

The MSC is working with our primary collaborator in the SMT program, DFO/CCG, to explore the possibility of adapting its automated voice technology used for Weatheradio to be used for the CCG’s marine broadcast system, in order to improve its efficiency. The MSC is also taking an active role in supporting E-navigation strategies that are being led by DFO/CCG, which will lead to new solutions for marine weather information access and display that are compatible with industry needs and expectations, domestically and internationally. These solutions will go beyond dissemination strategies, and will necessitate some reconsideration of the nature and formats of marine weather information entirely. To that end, the MSC is conducting a small project to convert marine forecasts into an xml format. Such a format opens the door to many innovative ways to disseminate the information, not only by the MSC but also by third parties.


Timeline
Deliverable(s)
Responsible Party
October 2011
New service agreement with a telecommunications company established in Atlantic Canada to enable the modernization of Atlantic Weatheradio network
DG, WEPS
July 2012
WES service strategy approved at ADM level
DG, WEPS
March 2013
Draft WES dissemination strategy tabled at DG leads table
DG, WEPS
2011-15
MSC contributes to DFO/CCG development of marine E-navigation initiative
DG, WEPS

Return to Table of Contents

Annex 1: Issues, Indicators and Methods

Evaluation Issues, Indicators and Methods (text description)
IssueIndicatorsMethods
1. Is there a continued need for Services to Marine Transportation?
  • Evidence of / views on societal/ environmental need for program
  • Document review
  • Literature review
  • Open-ended interviews
2. Are Services to Marine Transportation aligned with federal government priorities?
  • Program objectives correspond to recent/current federal government priorities
  • Program objectives are aligned with current departmental strategic outcomes
  • Document review
3. Is the program consistent with federal and departmental roles and responsibilities?
  • Program mandate aligned with federal government jurisdiction
  • Document review
  • Key informant interviews
3. Is the program consistent with federal and departmental roles and responsibilities?
  • Evidence of / views on legitimacy, appropriateness and necessity of departmental roles, responsibilities and accountabilities with respect to marine transportation
  • Document review
  • Open-ended interviews
  • Stakeholder/partner surveys
4. Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?
  • Existence of effective performance measurement strategy:
    • Performance indicators cover all outcomes
    • Performance indicators are linked to the outcomes being measured
    • Evidence of rigorous performance data collection and reporting activities
    • Satisfaction of decision makers with timeliness of data collection/reporting
  • Evidence of fully populated database(s)
  • Document review
  • Open-ended interviews
  • Performance data analysis
5. To what extent have intended outcomes been achieved as a result of the program?
  • Evidence of / views on achievement of intended outputs and outcomes
  • Output: Percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines
  • Output: Percentage of gale warnings issued at least 24 hours in advance of the event
  • Output: Percentage of ice warnings issued and verified prior to the event
  • Output: Frequency (percentage of times) that trained ice service specialists are provided as per partnership agreements
  • Outcome: Evidence of use of meteorological, oceanographic and ice information products and services by target users
  • Outcome: Views of program stakeholders on the accuracy, timeliness, accessibility / user-friendliness and use (by target users) of marine weather and ice information products and services
  • Outcome: Satisfaction of target end-users with the accuracy, timeliness, accessibility / user-friendliness and usefulness of marine weather and ice information products and services
  • Outcome: Evidence of reduced number of emergency interventions (by emergency response teams and/or the CCG) due to marine weather or ice
  • Outcome: Evidence of reduced unexpected delays encountered by commercial marine transports due to marine weather or ice
  • Evidence of / views on factors outside the direct purview of the program that may influence / have influenced the achievement of intended outcomes
  • Document review
  • Open-ended interviews
  • Performance data analysis
  • End-user questionnaire
6. Are Services to Marine Transportation achieving their intended outcomes in the most economical manner?
  • Views on whether good value is being obtained with respect to the use of public funds
  • Open-ended interviews
  • End-user questionnaire

7. Is the Services to Marine Transportation program undertaking activities and delivering products in the most efficient manner?

  • How could the efficiency of the program’s activities be improved?
  • Are there alternative, more efficient, ways of delivering the program?
  • Evidence of / views on whether the cost of producing program outputs is as low as possible
  • Document review
  • Financial analysis
  • International comparison
  • Open-ended interviews
  • End-user questionnaire

7. Is the Services to Marine Transportation program undertaking activities and delivering products in the most efficient manner?

  • How could the efficiency of the program’s activities be improved?
  • Are there alternative, more efficient, ways of delivering the program?
  • Views on how the efficiency of program activities could be improved
  • Views on whether there are alternative, more efficient ways of delivering program activities and outputs
  • Open-ended interviews
  • End-user questionnaire

Return to Table of Contents

Annex 2: Summary of Findings

Summary of Findings
Evaluation IssueAchievedProgress Made; Attention NeededLittle Progress; Priority for AttentionNot Applicable
Relevance:
1. Continued need for Services to Marine Transportation
   
Relevance:
2. Alignment of activities with federal government priorities and departmental strategic outcomes
   
Relevance:
3. Consistent with federal and departmental roles and responsibilities
   
Performance:
4. Collection of performance data and use of performance data to inform senior management / decision makers
   
Performance:
5. Achievement of intended outcomes
   
Economy and efficiency:
6. Acquisition of resources and management of program is conducted in an economical manner
   
Economy and efficiency:
7. Efficient program delivery
   

Return to Table of Contents

Executive Summary

Introduction

This report presents the results of the Evaluation of Services to Marine Transportation (SMT) conducted by Environment Canada’s (EC’s) Audit and Evaluation Branch between September 2010 and May 2011. It addresses the activities that fall under the Department’s Program Activity Architecture (PAA) 2.2.2 sub-activity (SA).

Although the evaluation was conducted within the context of a joint review and evaluation, this report contains the findings of the evaluation component only, and addresses the standard evaluation issues pertaining to program relevance and performance. The period examined is from 2007-08 to 2010-11.

EC’s SMT SA provides marine users with information to support decision making for marine activities and warnings for weather-related threats to safety. It serves the needs of maritime communities, and supports Canada’s obligations in providing safety information for international marine traffic. EC’s SMT SA comprises two main activities: the provision of marine weather information and the provision of ice information.

Within EC, delivery of the SMT SA is performed by the Meteorological Service of Canada (MSC), the organizational branch with responsibility for producing and disseminating weather and environmental information, including that related to ice.1 The MSC also relies on the Atmospheric Science and Technology Directorate of the Science and Technology Branch (S&T Branch) as a key contributor to meteorological research and development, and the Chief Information Officer Branch for activities related to information management and information technology. The activities performed under the SMT SA also rely heavily on the broader base of weather prediction services provided by other groups within the MSC and the S&T Branch, as delivered under the Department’s PAA activity 2.1 – Weather and Environmental Services for Canadians.

The evaluation methodology included a review of relevant federal and departmental documents, as well as key informant interviews with program stakeholders and a questionnaire that was distributed to end-users.

Findings and Conclusions

The following represents a summary of the overall findings and conclusions of the evaluation.

Relevance

Multiple sources indicated that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions.

Findings from multiple sources indicated that the SMT SA is aligned with the legislated mandates of the involved departments, and with responsibilities assigned through agreements and other mechanisms for collaboration. Furthermore, the evidence indicates that the MSC is obligated by various acts and agreements to deliver SMT SA services, including in the context of ongoing northern development.

For the most part, evidence found that EC’s SMT SA services complement services in other government departments. However, concerns were expressed by EC and collaborating department representatives regarding the extent to which some areas of shared responsibility are clearly articulated. There is additional concern that the Department’s ability to deliver on its responsibilities is vulnerable, as these services are reliant on key partners or collaborators.

In addition, the SMT SA is aligned with and supports federal government priorities related to health and safety, the economy, infrastructure and sovereignty, as well as departmental strategic outcomes and priorities related to ensuring that Canadians are equipped to make informed decisions on changing weather, water and climate conditions.

Key informants noted that one area of expanding interest for the federal government, and an area in which the MSC will likely provide increased marine weather and ice services, is northern development. The MSC has been resourced to undertake new approaches to providing services to Arctic communities and industries under a newly established Meteorological Areas (METAREAs) Initiative that is funded under this SA. However, as this initiative had only begun at the outset of the evaluation, it was not included in the analysis and will be evaluated by EC at a later date.

Performance

The evaluation found evidence of a strong commitment to performance measurement, supported by the MSC’s ISO 9001 Quality Management System. Considerable data collection is occurring, and program management appears to recognize the value of performance measurement.

Although the activities that comprise the SMT have existed within the MSC since its origins 140 years ago, the grouping of the provision of marine weather information and ice information as one PAA element, known as the SMT, was just recently introduced with the 2010-11 PAA. As such, the existence of a Performance Measurement Strategy that presents a logic model for the PAA element and a cohesive Performance Measurement Framework (PMF) for the entire element does not yet exist, although efforts are being made in this area.

Nevertheless, the evaluation found that the SMT SA was able to provide data and targets on key outputs that were identified for the purpose of the evaluation, such as the delivery of marine weather warnings and ice forecasts and charts, and that, furthermore, it is performing at or near the vast majority of these targets. With regards to the availability of performance measurement related to outcomes, data were also available for the outcomes identified for the purpose of the evaluation, although baseline data that would support an understanding of how performance has changed over time was limited.

The evaluation found evidence of strong performance regarding outcomes related to usefulness and ease of understanding of weather and ice information products. There is a very high incidence of mariners accessing and making use of the marine weather and ice information provided by EC, and overall satisfaction with the products and services was high. Marine and ice information is disseminated to mariners with a broad range of methodologies. Although the EC Weatheroffice website was identified as the most frequently used source of marine weather information, evidence found that most mariners relied on more than one source or technology for their information needs, and further, that users of marine information are a diverse group who include both those who rely on receiving information in the same manner in which it has been provided for many years, as well as those who would like to see further enhancements that leverage advanced technological capabilities. This presents a challenge for EC from an efficiency perspective, as it makes it very difficult to find cost savings to fund new advancements from the removal of historic products.

The SMT SA operates on a budget of approximately $18.5 million per year, and spending on these activities has been relatively stable over the four years of the study period. EC’s SMT is not, however, a self-contained function, but rather is able to deliver on its outcomes by relying both on the broader underlying capacity of the MSC for weather and environmental services as well as its important relationships with its key partner, the Canadian Coast Guard (CCG), and many other collaborators and stakeholders, including Transport Canada, the Canadian Space Agency, the World Meteorological Organization, and the U.S. National Ice Center, among others. These relationships and the cooperative, collaborative manner in which the Department operates in this area, supports EC’s ability to deliver marine transportation services in a cost-effective manner. The relationship with the CCG appears strong, for the most part, although there are areas where roles and responsibilities could be clarified. In particular, clarifying roles related to servicing weather buoys and providing storm surge warnings were identified as areas lacking clarity. Addressing areas of ambiguity will have the added advantage of ensuring that the inherent risk associated with reliance on another party is minimized.

Recommendations

The following recommendations are based on the findings and conclusions of the evaluation. The evaluation recommendations are directed to the Assistant Deputy Minister (ADM), MSC, in light of the ADM’s responsibility for the overall management of SMT activities.

Recommendation 1

During the conduct of this evaluation, the evaluation team was able to rely on a variety of performance data from the SMT SA’s initial draft 2010-11 PMF to develop a preliminary assessment of various aspects of the SMT SA’s performance. Working with this, the evaluation team developed a set of outputs and outcomes in order to support a preliminary assessment of performance. As noted in the report, there was some concern expressed by SMT management at the onset of the evaluation that the outputs and outcomes used for the evaluation may not be adequately aligned with SMT services and activities, and that management may not be able to collect the required data. It is notable that the program was able to produce data across these outputs and outcomes, and this does represent a significant step toward ongoing performance reporting. Additionally, it is clear that the SA recognizes the importance of formalized performance measurement, through ongoing development of the SMT SA’s PMF. However, several gaps in performance reporting still remain, including a more thoroughly validated set of performance indicators across expected outputs and outcomes, and a reporting strategy that integrates these data sources into a common reporting tool. It is therefore recommended that the ADM MSC, using the evidence presented in this report as a baseline for performance reporting, continue to develop the PMF, including a logic model, expected program outputs and outcomes, and a strategy to collect and report on this information. Attention should be given to the performance reporting requirements of organizations such as the CCG.

Recommendation 2

The services provided by SMT are mandated by a range of obligations set out in both international and domestic acts and agreements. Furthermore, these acts and agreements link SMT to a variety of federal and international collaborating organizations, and have clearly formalized several key areas of SMT service provision (such as the deployment of ice specialists onboard CCG ships). Although obligations clearly establish the need for SMT to provide maritime weather and ice services, there are nevertheless potential risks to SMT if there are changes in the level or type of service provided, or in the financial and human resource commitments by key collaborating departments. During this evaluation, there was particular concern expressed regarding clarifying the relationship with the Department of Fisheries and Oceans (DFO) / CCG in areas such as the maintenance and servicing of buoys, and the provision of storm surge warnings in various regions. It is therefore recommended that the ADM MSC develop a strategy to identify and address risks posed to the SMT SA due to any areas of uncertainty in its relationships with collaborating departments. As a starting point, this strategy should include an approach to formalize the relationship with DFO/CCG regarding servicing weather buoys and providing storm surge warnings.

Recommendation 3

It was frequently noted that the SMT SA must use an increasingly wide range of communications technologies to disseminate meteorological information across all potential users, and that program management is limited in the number of media that can be reduced or removed, as it may pose safety risks to end-users that rely on these specific forms of media. Given the growth in available potential methods of dissemination in relation to current resources, it is therefore recommended that the ADM MSC establish a strategy to manage the evolution of dissemination strategies for marine weather and ice information, and to examine the consequences of reducing or removing current dissemination mechanisms. This strategy should be based, to the extent possible, on the documented needs of SMT’s maritime clients and should include cost estimates for each proposed new approach. A process should be established to ensure that the strategy is reviewed and revised on a regular basis.

Management Response

Recommendation 1


The MSC agrees with the recommendation.


Management Action

The MSC already has a PMF in place for the SMT program (PAA sub-activity 2.2.2), and has been updating the program outputs and outcomes as well as strategies to collect and report on this information, for several years. Updates to the expected outcomes and outputs for the 2012-13 reporting cycle are awaiting EC management approval, and are being submitted to Treasury Board Secretariat in the summer of 2011.

As part of the development of a performance strategy for the METAREAs marine services (a new initiative that falls under the SMT program), a logic model and PMF were developed during 2010-11. Through this exercise for the METAREAs, there is transferable information that will be used in the development of a logic model for the SMT program. The MSC is committed to delivering on a logic model and updating its Performance Measurement Strategy by March 2013, after sufficient consultation with its clients and partners.

In the initial IISPA (Ice Information Services Partnership Agreement) between the MSC’s Canadian Ice Service and the CCG, a logic model and performance measures were developed. Reporting on all of them became challenging for both, and thus, in the re-negotiated IISPA, the number and type of performance measures have been reduced and focused significantly, with the intention to refine them in the coming year.

Pre- and post-season meetings with the CCG and clients result in requests for new products, and reductions and/or changes in other products. These are usually captured in the minutes of these meetings and actioned by either party.


Timeline

Deliverable(s)

Responsible Party
March 2012
Logic model for SMT program finalized
Director General (DG), Weather and Environmental Prediction Services (WEPS)
March 2013
Updated PMF outputs, outcomes, and performance data collection and reporting strategy finalized
DG, Business Policy (or DG, WEPS)

Recommendation 2


The MSC agrees with the recommendation.


Management Action

The MSC enjoys benefits and synergies from its collaborative arrangements, but also acknowledges the risk posed by uncertainties with respect to its relationships with some collaborating parties, including other government departments. This risk has been identified in the Weather and Environmental Services (WES) Risk Management Profile, although the risk statement does not specifically note the weather buoy and storm surge warning elements associated with the relationship with DFO/CCG. This dimension of the collaboration risk will be more fully reflected in the next update to the Risk Management Profile. The MSC’s Marine and Ice Program Management Board will be tasked with developing strategies to analyze and mitigate these risks, and to present a strategy to MSC senior management before the end of 2011-12.


Timeline
Deliverable(s)
Responsible Party
December 2011
Reflect collaboration risks of SMT program in the WES 2012-13 Risk Management Profile
DG, Business Policy
March 2012
Analysis and strategy document dealing with collaboration risks in the SMT presented to MSC management committee
DG, WEPS

Recommendation 3


The MSC agrees with the recommendation.


Management Action

There is a risk, when in the business of information dissemination, of chasing the newest technologies (e.g., social networking) without fully understanding client needs and the effectiveness of the technologies in meeting these needs. The MSC’s dissemination strategies take into consideration a number of factors, including client needs, technological opportunities, synergies among service areas, partnering opportunities (e.g., with CCG Marine Radio) and fiscal realities. The needs of mariners have had a major influence on our dissemination systems--most notably the design of the MSC’s Weatheradio network, which has a significant mariner audience. We recognize that the marine community has unique requirements (e.g., often not having access to cell phones and the Internet when at sea), hence some solutions for public weather information dissemination are inadequate for the marine and ice program.

Strategies for dissemination should be based on broader service strategies, and the MSC is in the process of documenting a new service strategy. Once approved, this strategy will provide guidance with respect to many service directions, including dissemination strategies. In the meantime, MSC dissemination experts continue to work with the telecommunications industry to stay abreast of the latest technological advances, while client relations personnel, including those serving the SMT program clients, interact with client organizations to ascertain how their information–access needs are changing. One example of an outcome resulting through these processes is the redesign and reconfiguration of the Weatheradio network in Atlantic Canada, which is currently under way. These changes will see more region-specific programming from each transmitter location, hence reducing the program cycle (the length of which has been a frequent complaint from mariners).

The MSC is working with our primary collaborator in the SMT program, DFO/CCG, to explore the possibility of adapting its automated voice technology used for Weatheradio to be used for the CCG’s marine broadcast system, in order to improve its efficiency. The MSC is also taking an active role in supporting E-navigation strategies that are being led by DFO/CCG, which will lead to new solutions for marine weather information access and display that are compatible with industry needs and expectations, domestically and internationally. These solutions will go beyond dissemination strategies, and will necessitate some reconsideration of the nature and formats of marine weather information entirely. To that end, the MSC is conducting a small project to convert marine forecasts into an xml format. Such a format opens the door to many innovative ways to disseminate the information, not only by the MSC but also by third parties.


Timeline
Deliverable(s)
Responsible Party
October 2011
New service agreement with a telecommunications company established in Atlantic Canada to enable the modernization of the Atlantic Weatheradio network
DG, WEPS
July 2012
WES service strategy approved at ADM level

DG, WEPS
March 2013
Draft WES dissemination strategy tabled at DG leads table
DG, WEPS
2011-2015
MSC contributes to DFO/CCG development of marine E-navigation initiative
DG, WEPS

   


1. The Canadian Ice Service (CIS), a division of the MSC, is responsible for delivery of the activities related to ice information. Excluded from the scope of the evaluation are the services provided by the CIS relating to pollution monitoring, prevention and enforcement. These activities relate to the PAA element 3.1.2.2 – Marine Pollution.

Return to Table of Contents

Tables and Figures

Tables and Figures

Table 1: Annual Financial Expenditures for 2.2.2 Services to Marine Transportation

Description: Table 1 displays the salary, O&M, Capital, VNR –Salary and VNR –O&M expenditures for the individual marine and ice components of the SMT program, as well as the total, for the four years from 2007-08 to 2010-11. 2010-11 also includes the expenditures for the Metareas initiative.

Return to the table


Table 2: SMT Service by Region

Description: Table 2 identifies which program services (i.e., short-term forecast, weather warnings, extended forecast, technical marine synopsis, marine weather statement, wave height forecast, NAVTEX forecast, ice hazard bulletins, recreational boating marine forecast, MAFOR forecast) are delivered in each of the six marine regions.

Return to the table


Table 3: Services to Marine Transportation Program Results Chain

Description: Table 3 presents the program results chain for the SMT Program, identifying the activities, outputs and target clients, as well as the outcomes for the program and the higher level departmental strategic outcomes that the program contributes toward.

Return to the table


Table 4: Key Informant Interviewees: Long-form Interview Guide

Description: Table 4shows the total number of interviews completed for each key informant type. The total number of interviews completed is also included in the table.

Return to the table


Table 5: End User Type: Questionnaire

Description: Table 5shows the total number of interviews completed for each category of end-user (i.e., recreational mariner/sport fisher, professional mariner, professional fish harvester, ship manager or operator, other). The total number of interviews completed with end-users is also included in the table.

Return to the table


Table 6: Rating Symbols and Significance

Description: Table 6 displays the scale used to rate the degree to which relevance and performance has been achieved by the program. These ratings include: Achieved; Progress Made, Attention Needed; Little Progress, Priority for Attention; and Not Applicable. The tilda (~) symbol denotes that the outcomes achievement ratings have been based solely on subjective evidence.

Return to the table


Table 7: SMT Linkages by Subject Area

Description:Table 7 lists the organizations involved with various aspects or subject areas of the SMT program.

Return to the table


Table 8: Performance Measurement Data Availability

Description: Table 8 lists the indicators associated with various outputs and outcomes of the SMT SA, and describes the data that is collected against each of these indicators, including the frequency/availability of the data (e.g., monthly or quarterly data available, no data available).

Return to the table


Table 9: Detailed List of Outputs and Results

Description: Table 9 presents the data collected against the SMT indicators, as listed in table 8, and provides a summary of the results of the data related to the performance of the SMT SA.

Return to the table


Figure 1: Distribution of Marine Weather Sources used by Respondents

Description: Figure 1 is a bar chart that presents the number of respondents from the survey of end users using each of a variety of sources for marine weather information. It also includes the number of respondents who indicated that a particular source was their main source.

Return to the figure


Figure 2: Rating of the Marine Weather Service

Description: Figure 2 is a bar chart that presents the distribution of ratings by survey respondents on various characteristics of the weather service (accuracy, usefulness, ease of understanding, timeliness and accessibility), on a scale of 1 to 5 where 1 means “poor” and 5 means “excellent”.

Return to the figure


Table 10: Transportation Safety Board – Summary Table 2005-2009

Description: Table 10 provides a summary of key statistics from the Transportation Safety Board for the five years from 2005 to 2009, including the number of reports which occurred during the year, the number of incidents where weather or ice conditions was a factor, the number of reports where service provided by the MSC was accurate and a brief summary of the type of incidents reported.

Return to the table


Table 11: Annual Financial Expenditures for 2.2.2 Services to Marine Transportation (identical to Table 1)

Description: Table 11 displays the FTEs, salary, O&M, Capital, VNR –Salary and VNR –O&M expenditures for the individual marine and ice components of the SMT program, as well as the total, for the four years from 2007-08 to 2010-11. 2010-11 also includes the expenditures for the Metareas initiative.

Return to the table


Table 12: Year over Year Changes in Expenditures

Description: Table 12 presents the year over year changes in spending for the the FTEs, salary, O&M, Capital, VNR –Salary and VNR –O&M expenditures for the individual marine and ice components of the SMT program, as well as the total, for the four years from 2007-08 to 2010-11. 2010-11 also includes the expenditures for the Metareas initiative.

Return to the table


Table 13: Percentage distribution of resources over last four fiscal years

Description: Table 13 presents the percentage distribution of expenditures among the three branches (MSC, S&T, CIOB) for the marine component and the ice component of SMT for the 4 years from 2007-08 to 2010-11. The total expenditure distribution by branch is also presented for each year.

Return to the table


Figure 3: Relative Spending on Meteorological Services

Description: Figure 3 displays a bar graph on the relative spending on meteorological services, sorted by country. For each country the graph indicates per capita expenditures on meteorological services and dollars per square kilometre on meteorological services.

Return to the figure


Annex 1: Evaluation Issues, Indicators, Data Sources

Description: Annex 1 depicts the evaluation matrix, which describes the questions addressed in the evaluation. The table also provides the indicators used to measure achievement and the proposed sources and/or methods that were used to obtain evidence for each indicator.

Return to the table


Annex 2: Summary of Findings

Description: Annex 2 provides a table displaying a summary rating for each evaluation question that was addressed, sorted by relevance, performance, and economy and efficiency. These ratings include: Achieved; Progress Made, Attention Needed; Little Progress, Priority for Attention; and Not Applicable.

Return to the table

Return to Table of Contents

Date modified: