This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Skip booklet index and go to page content

Evaluation of the Services to Marine Transportation Sub-activity

Findings

The findings of this evaluation are presented below by evaluation issue (relevance and performance) and the related evaluation questions. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question based on a judgement of the evaluation findings. The rating symbols and their significance are outlined below in Table 6. A summary of ratings for the evaluation issues and questions is presented in Annex 2.

Table 6: Rating Symbols and Significance (text description)
SymbolSignificance
AchievedThe intended outcomes or goals have been achieved or met
Progress Made; Attention NeededConsiderable progress has been made to meet the intended outcomes or goals, but attention is still needed
Little Progress; Priority for AttentionLittle progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis
N/AA rating is not applicable
~Outcomes achievement ratings are based solely on subjective evidence.

4.1 Relevance

Evaluation Issue: Relevance


Overall Findings: Relevance

Multiple sources indicate that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions. Findings from multiple sources also indicate that activities and services offered under the SMT SA are consistent with the legislated mandates of the involved departments and with responsibilities assigned through agreements and other mechanisms for collaboration. The need for SMT services was particularly apparent given the evidence indicating that the MSC is obligated by various acts and agreements to deliver SMT SA services, including in the context of ongoing northern development. In addition, it was noted by key informants that the SMT SA is aligned with and supports federal government priorities and departmental strategic outcomes.

Some concerns were expressed regarding the extent to which some areas of shared responsibility are clearly articulated. Furthermore, there is additional concern that the Department's ability to deliver on its responsibilities is vulnerable, as these services are reliant on key collaborating departments and agencies.


4.1.1 Continued Need for the Program

Continued Need for the Program
Evaluation Issue: RelevanceIndicator(s)MethodsRating
1. Is there a continued need for SMT?
  • Evidence of / views on societal/ environmental need for program
  • Document review
  • Literature review
  • Key informant interviews
Achieved

Multiple sources indicate that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions. The continued need is particularly apparent given that the MSC is obligated by various acts and agreements to deliver SMT services, including in the context of ongoing northern development. This need was also recognized by all key informants interviewed as part of the evaluation, with some indicating that it is fundamental to supporting the health and safety of mariners, instrumental in supporting fishing and other marine industries, and important in the context of northern climate change and economic development, as the level of maritime traffic in the north has been increasing due to longer navigation periods in Arctic waters.

The following extract outlines the key reasons for the provision of ice and marine weather information as a public service:

As a northern nation, Canada must know about its ice environment and the impacts it has on human activities. As a trading nation dependent on marine transportation, we must deal with the hazards that the constantly shifting ice presents to ships. Three of the five largest Canadian ports are affected by ice for a large part of the year and rely on ice information to stay active and viable. It is believed that because of the widespread public impact that ice has on broad sectors of the economy and because of the safety hazards it presents to the marine public, there has been little question that providing ice information is an essential federal government service. The public benefits are too widely dispersed to sustain a viable private sector service except in certain niche areas, such as site–specific information for offshore oil platforms. Like all other northern nations, Canada provides an ice information service as a public benefit.21

The need for services offered by SMT is expected to continue and even increase. The CCG Fleet Annual Report for 2007-2008 noted that "Climate change, notably in the Arctic, is already extending the duration of the commercial shipping season, and may further intensify the demand for marine science and other activities."22

The document review indicates that the provision of SMT SA services should not be considered an optional service for EC. Rather, EC is obligated by a variety of acts and agreements to provide these services, both in partnership with other departments and directly to Canadians. For example23:

  • Canada is a signatory to the United Nations Convention on the Law of the Sea (UNCLOS). Article 43 of UNCLOS charges states adjoining straits to cooperate in aiding navigation and in preventing, reducing and controlling pollution from ships. The provision of SMT thus enables Canada to meet its international obligations respecting marine safety.24 Similarly, Canada is a signatory to the International Convention for the Safety of Life at Sea. Chapter V of this convention lists certain navigation safety services that should be provided by governments, including the maintenance of meteorological services for ships, the ice patrol service, routing of ships, and the maintenance of search and rescue services.25
  • Canada has committed to the IMO to provide meteorological information and navigational data to facilitate the safe management of marine traffic in two well-defined Arctic areas that are substantially within Canadian territory. The areas include Canadian Arctic waters, such as the Northwest Passage, as well as adjacent waters north of Alaska and along part of the western coast of Greenland.
  • As noted in the Department of the Environment Act, the Minister of the Environment has legislative authority for all matters relating to meteorology, water, and boundary waters, and for the provision of environmental information to Canadians.26 As such, EC is federally mandated to provide authoritative, quality meteorological and environmental warnings, forecasts, services and information for the overall betterment of Canada.27
  • Since 1995, the CIShas entered into consecutive five–year MOUs with the CCG to provide information related to ice conditions in order to assist that agency in meeting its mandate.
  • A Senate Report on the CCG in 2009 (and by implication the CIS) recommended that Canada develop a much stronger year-round national presence and enforcement capability, to demonstrate commitment to controlling the Northwest Passage, protecting Canadian interests and Canada's northern residents, and making the waterway a safe and efficient shipping route.28
  • Budget 2010 provides $9.2 million over two years to EC and $2.2 million over two years to DFO to deliver meteorological and navigational services, respectively, in the Arctic, in order to meet Canada's commitments to the IMO regarding these areas.29

The continued need for the services of the SMT SA was also recognized by all key informants interviewed as part of the evaluation. A plurality of respondents indicated that they thought the SMT SA is fundamental to supporting the health and safety of mariners, and that it is instrumental in supporting the fishing and other marine industries. Furthermore, approximately one in four key informants indicated that there are no alternative providers of these SMT SA services and that these services are a public good. Approximately one in four indicated that the SMTSA is particularly important in the context of northern climate change / economic development, as the level of maritime traffic in the north has been increasing due to longer navigation periods in Arctic waters. Other comments included the fact that the SMT SA is essential for the activities of other federal departments (examples include supporting the CCG and the Department of National Defence [DND] in search and rescue operations and environmental protection), and that SMT SA supports the development of private sector organizations that use generalized EC data to provide a level of detailed information (e.g., for offshore drilling and support to northern communities).

4.1.2 Alignment with Federal Government Priorities

Alignment with Federal Government Priorities
Evaluation Issue: RelevanceIndicator(s)MethodsRating
2. Are Services to Marine Transportation aligned with federal government priorities?
  • Program objectives correspond to recent/current federal government priorities
  • Program objectives are aligned with current departmental strategic outcomes
  • Document review
Achieved

The SMT SA is aligned with and supports federal government priorities (e.g., health and safety, the economy, infrastructure and sovereignty) as well as departmental strategic outcomes and priorities.

Alignment with Federal Government Priorities

In terms of recent priorities, the Government of Canada has been active in promoting and defending Canada's sovereignty over Arctic waters. The 2010 Speech from the Throne contained the following commitment:

Our Government will continue to vigorously defend Canada's Arctic sovereignty. It will continue to map our northern resources and waters. It will take action to increase marine safety and reduce pollution from shipping and other maritime traffic.30

Arctic sovereignty was a theme of the 2007 Speech from the Throne as well, which highlighted a need for improved patrol, surveillance, and environmental research capabilities.31SMT services such as supporting patrol and surveillance capacity are essential to furthering the goals of establishing and defending sovereignty, and to the ongoing development of Canada's North.

Budget 2010 contained several provisions related to the SMTSA. Most directly, the Budget provided $9.2 million to ECand $2.2 million to DFO, over two years, to deliver meteorological services in the Arctic to help meet Canada's commitments to the IMO. Other Budget provisions that indicate the continued importance of SMT include $497 million over five years for the CSA's RADARSAT Constellation Mission (which has marine and ice applications); and $27.3 million over five years to aid the CCGin ensuring maritime safety, as well as $175 million to the CCG for the repair and refit of existing vessels and the purchase of new vessels.32

Key informants also indicated that the SMT SA is aligned with government priorities. Nearly half of the key informants indicated that the SMT SA helps support the Canadian economy and efforts for economic development (particularly in the North), and that the SMTSA contributes in key ways to ensuring the health and safety of Canadians. Several key informants also noted that the SMTSA is important for federal efforts to enhance infrastructure in the North.

Alignment with Departmental Strategic Outcomes

SMTinformation products (e.g., ice and marine weather warnings) are designed to help people across Canada respond appropriately to marine and ice weather conditions. The newly developed Marine and Ice Program Management Board (MIPMB) is tasked with managing the activities of EC's SMT as identified within EC's PAA.33,34 The delivery of the SMTSA supports the departmental strategic outcome "Canadians are equipped to make informed decisions on changing weather, water and climate conditions." Weather prediction activities also align with one of three strategic outcomes in EC's Sustainable Development Strategy 2007-2009: "Weather and Environmental Services: Weather and environmental predictions and services reduce risks and contribute to the well-being of Canadians."35

Furthermore, key informants also noted that SMT is aligned with departmental priorities. For example, approximately one in five key informants indicated that the SMT SA helps support departmental efforts to protect the environment, such as from oil spills to water bodies, and that it contributes to the safety and security of marine transport. Several key informants also indicated that the SMTSA helps support the integrated decision making of EC's WESBoard.

4.1.3 Consistency with Federal and Departmental Roles and Responsibilities

Consistency with Federal and Departmental Roles and Responsibilities
Evaluation Issue: RelevanceIndicator(s)MethodsRating
3. Are Services to Marine Transportation consistent with federal and departmental roles and responsibilities?
  • Program mandate aligned with federal government jurisdiction
  • Evidence of / views on legitimacy, appropriateness and necessity of departmental roles, responsibilities and accountabilities with respect to marine transportation
  • Evidence of duplication and/or complementary activities with other government departments (OGDs)
  • Document review
  • Key informant interviews
Achieved

The provision of the SMT SA is consistent with the legislated mandates of the involved departments and with responsibilities assigned through agreements. For the most part, comments from key informants focused on the SMT SA services' complementary nature with OGDs. Some concerns were expressed by EC and collaborating organization representatives regarding the extent to which some areas of shared responsibility are clearly articulated.

The roles and responsibilities for the provision of services to marine communities and marine industries are spread across EC, DFO and TC. Generally, the commitments of the various departments with respect to providing services to aid marine transportation are as follows:

  • EC has a responsibility to provide meteorological information of interest to Canadians, for general and emergency management purposes.
  • DFO has a responsibility to provide ice management, channel maintenance, and navigational information. Although these services may be augmented by the assistance of other departments, DFO is authorized to conduct the research and information production activities that are necessary to meet its requirements.
  • TC has a responsibility to regulate marine transportation activities in order to promote the health and well-being of those involved in marine transportation.

EC's mandate for federal involvement in providing services to aid marine transportation is set by sections 4 and 5 of the Department of the Environment Act, which assigns to the Minister of the Environment jurisdiction over water and meteorology issues and responses, as well as the provision of environmental information to Canadians and the development and coordination of environmental programs.

Section 6 of the Emergency Management Act requires all departments to develop and implement emergency management plans in relation to the risks associated with their area of responsibility. In terms of the coordination of efforts between levels of government in the case of meteorological emergencies, EC is responsible for issuing marine warnings; flood warnings are issued by provincial governments.36

The following table sets out the various linkages with other organizations, for EC's SMT, by subject area.

Table 7: SMT Linkages by Subject Area37 (text description)
Subject AreaLinkages
Marine safety and navigation in iceTC, International Hydrographic Organization, IMO, Shipping Federation of Canada, U.S. Coast Guard, International (North Atlantic) Ice Patrol, International Navigation Association
Marine meteorology and ice information servicesWMO/JCOMM Expert Team on Sea Ice, U.S. National Ice Center, other national ice services, International Ice Charting Working Group, others
Satellite missions for ice and marine surveillanceCSA, European Space Agency, Natural Resources Canada (NRCan)–Canada Centre for Remote Sensing (CCRS), NOAA, North American Space Agency (NASA), Japanese Space Agency
Satellite data processing and satellite-based operational servicesCSA, European Space Agency, NRCan–CCRS, NOAA, NASA
Arctic transportation and sovereigntyDFAIT
Ice climatology and climate change researchClimate Research Branch–MSC, Canadian Foundation for Climate and Atmospheric Sciences, Canadian Collision Industry Forum, National Snow and Ice Data Center, ArcticNet, Canadian Arctic Shelf Exchange Study
Ice engineering and environmental designNational Research Council–Canadian Hurricane Centre, C-CORE, other industry

For the most part, comments from key informants focused on the complementary nature of SMT SA services with OGDs. For example, nearly half of the key informants noted that the SMT provided by EC do not duplicate private sector endeavours. Private sector endeavours tend to build upon publicly provided information, and are frequently aligned with a specific set of users (e.g., skiers, offshore oil and gas industry). Approximately one in four also noted that the SMTSA complements and supports the work of other organizations (e.g., DFO/CCG, TC, International Joint Commission, CSA, U.S. National Ice Service).

However, key informants pointed to the potential risks associated with the extent to which the SMT SA relies on collaborators to meet its program objectives (e.g., DFO and NAV Canada). The risk exists that, if these relationships were terminated, EC may be unable to deliver on certain services until other suitable collaborators could be identified. If business relationships with collaborators or third parties are not well managed, or if there is a failure on the part of collaborators, this could negatively affect program or service delivery. This issue is discussed further in the section related to efficiency and economy.

4.2 Performance

Evaluation Issue:


Overall Findings: Performance

Overall, the document review and key informant interviews indicate that there has been notable effort to ensure that the performance measurement data collection and reporting system provides information on key outputs and outcomes, and that these are currently managed through the early PMF development and ISO 9001 Quality Objectives. The main area for improvement at present would appear to be developing comprehensively managed reporting through methods such as an annual report, in order to ensure that the wealth of data are reported in a consistent and unified manner. There is also some room for improvement with respect to the completeness and regional consistency of data and the use of performance data to inform senior management / decision makers.

A review of the evidence found that the SMT SA is able to provide data and targets on its key outputs, and that the SMT SA is performing at or near the vast majority of these targets. With regards to performance against outcomes, evidence of strong performance regarding access to and use of weather and ice information products exists. Data were more limited for the outcomes related to a reduced need for emergency interventions, and reduced unexpected delays by commercial marine transports due to marine weather or ice, but some information was available nonetheless. However, given the recent establishment of the SA and outcomes, no historic data exist to measure the degree to which these outcomes have seen changes over time. Data gathered during this evaluation can be used for future refinement of a PMF and, where appropriate, as a baseline against which future performance can be measured.

The evidence related to efficiency and economy is preliminary, but points toward an SA that is taking steps to ensure that SMT SA inputs are appropriate and costs of outputs are minimized. Areas for improvement and future challenges include clarifying roles and responsibilities with DFO/CCG related to servicing buoys and storm surges, and exploring possible cost savings opportunities associated with a better use of emerging technologies.


4.2.1 Collection and Application of Performance Data

Collection and Application of Performance Data
Evaluation Issue: PerformanceIndicator(s)MethodsRating
4. Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?
  • Existence of effective performance measurement strategy:
    • Performance indicators cover all outcomes
    • Performance indicators are linked to the outcomes being measured
    • Evidence of rigorous performance data collection and reporting activities
    • Satisfaction of decision makers with timeliness of data collection/reporting
    • Evidence of fully populated database(s)
  • Evidence of reporting of performance data to senior management / decision makers
  • Document review
  • Key informant interviews
  • Performance data analysis
Progress Made; Attention Needed

Overall, the document review and key informant interviews indicate that there has been notable effort to ensure that the performance measurement data collection and reporting system provides information on key outputs and outcomes, and that these are currently managed through the draft PMF and ISO 9001 Quality Objectives. The SMT SA is able to provide data and targets on its key outputs; and performance data exist, although in a more limited manner, on its identified outcomes. The main area for improvement is developing comprehensively managed reporting in order to ensure that the wealth of data being collected is reported in a consistent and unified manner, so as to inform senior management and decision makers. There is also some room for improvement with respect to the completeness and regional consistency of data.

The SMTSA has many of the key components for the development of an effective PMF, and evidence shows that attention has been given to this area. At a broad level, the evaluation found that the SMT SA has been generally effective at collecting performance measurement data. The level of available data is perhaps not surprising, given that the main program components of the SMTSA have been in place for many years. There are regular performance reports for the CIS, CCG icebreaking, and public and marine warnings, and the CIS/CCG partnership agreement outlines performance measures. Furthermore, there was a PMF under development during the evaluation to specifically address the SMT SA, as well as performance measures established in the Quality Management System (QMS) Manual (as part of the ISO 9001 Quality Objectives),38 and a Marine Forecast Verification Project aimed at improving the quality of marine forecasts and warnings. Details are as follows:

  • A draft PMF for the SMT SA for 2010-2011 outlines a framework for measuring several outputs and outcomes, although targets and timelines had not been established for all of the indicators.39
  • The Ice Information Services Partnership Agreement (IISPA) sets out performance measures for the CIS's provision of services to the CCG.40 The performance measures set out in this agreement are collected, for the most part, in the Canadian Ice Service Monthly Performance Measurement Report, which delivers on the agreements' intended monthly and quarterly reports. However, there were a few measures missing from the report, and several related to quality objectives that were supposed to be collected through an annual survey. Data that were to be collected through this annual survey do not appear to exist, due to restrictions on survey work that could be classified as public opinion research.
  • The WESQMS Manual indicates that efforts are being made to develop and track performance measures for: marine warnings, forecasts, and information products; and ice information products (ice and iceberg forecasts, warnings, and charts) as well as ice service specialists.41
  • The Marine Forecast Verification Project shows that efforts are being made to develop a nationally consistent program that verifies the performance of marine forecasts and marine warnings across Canada, to complete the performance measurement initiatives associated with all of its primary forecast programs. It will "fill an information gap that has existed for some time" and "ensure that a detailed database of forecast and observational data is maintained.42

The evaluation found that data are available to some degree for all of the outputs and outcomes identified in the draft PMF43 for the SA and those identified in the evaluation plan.44,45 Most of the performance measures appropriately capture the outcomes that are being measured, although in some cases the data do not align precisely or the periods of coverage are inadequate or have gaps. Table 8 summarizes the data collected against the identified outputs and outcomes.

Table 8: Performance Measurement Data Availability (text description)
IndicatorsData collectedAvailability
Output – Ice information products: Percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines
  • Percentage of ice forecast bulletins that were delivered on-time by region
  • Percentage of ice charts that were delivered on-time by region46
– Monthly data
Percentage of ice warnings issued and verified prior to the event
  • No data
– No data
Output – Ice information field services: Frequency (percentage of times) that trained ice service specialists are provided as per partnership agreements
  • Person days delivered as a percentage of person days requested47,48
– Monthly data
Output – Marine weather information products: Percentage of gale, storm, and hurricane-force warnings issued at least 24 hours in advance of the event
  • Percentage of gale, storm, and hurricane-force warnings with lead times of 24 hours or more49
– Quarterly data
Evidence of forecast accuracy
  • Probability of Detection (POD) – gale, storm, and hurricane force50
  • Forecast Accuracy Ratio (FAR) – gale, storm, and hurricane force51
  • Critical Success Index – gale, storm, and hurricane force52
– Quarterly national data and monthly regional data
Outcome – Access to and use of meteorological, oceanographic, and ice information products and services by target users
  • Percentage of mariners using marine weather forecasts, and daily frequency of use
  • Frequency with which mariners change sailing plans due to forecasts and percentage following recommended ice route
  • Use and source of weather information
  • Frequency of use of Port Meteorological Officer services, radio aids for marine navigation, and Navigational Telex (NAVTEX) and Marine Forecast (MAFOR)
– Survey conducted in 2004, skewed toward recreational mariners53,54
Outcome – Access to and use of meteorological, oceanographic, and ice information products and services by target users
  • Icebreaking services requested by region, country, service requested, vessel and cargo
  • Weatheroffice website and Really Simple Syndication visits for marine products

– Annual data55

– Annual data56

Outcome – Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice
  • Number of shipping accidents classified as ice damage
– 21 years of annual data are available from the Transportation Safety Board57
Outcome – Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice
  • Client delay due to ice conditions as a percentage of total service delays
– Annual data58
PAA Outcome – Marine communities have the weather, wave and ice information they need to operate safely and efficiently in Canadian waters: Target end-users' satisfaction with the accuracy, timeliness, accessibility / user-friendliness and usefulness of marine weather and ice information products and services
  • Level of understanding, usefulness, accessibility, timeliness and accuracy of weather information and reasons for low ratings
  • Importance of forecasts
  • Importance of daily ice charts and recommended ice route
  • Percentage of mariners reporting that the weather elements warning system meets needs
– Data from a 2004 survey59
PAA Outcome – Marine communities have the weather, wave and ice information they need to operate safely and efficiently in Canadian waters: Target end-users' satisfaction with the accuracy, timeliness, accessibility / user-friendliness and usefulness of marine weather and ice information products and services
  • Client engagement sessions concerning clients' issues and concerns with services
  • Received feedback during the Maritimes Region Marine Day from discussions and a survey
  • Received feedback during the North Atlantic Fish and Workboat Show from discussions and a survey

– 30 regional client engagement sessions in 200760

– Discussions with 80 visitors, and 27 surveys61

– Discussions with 280 visitors, and 59 surveys62

The key informant interview responses on the state of performance measurement for the SA generally align with the findings from the review of documents related to performance measurement. Generally, the opinion from key informants was that performance measurement has recently undergone, and is continuing to undergo, substantial improvements. Several individuals indicated that the ISO 9001 Quality Objectives were an important contributor to improvements in performance measurement. The Marine Verification Project was also noted as a contributor to improved quality of performance measurement information, as it collects data on the accuracy of marine weather products to support ongoing improvements.

Though the majority of interviewees suggested that performance measurement data are being used to inform decision–making processes, no documents were found in the document review that indicated decisions were being made based on collected performance measurement data. During the interviews, however, key informants did provide specific examples, including the following:

  • The Marine and Ice Program Board applies performance measurement information from end-users to guide priorities for operations (for example, extending the forecast period to five days).
  • Data are being used to improve forecasting. As data for verification of products such as wind warnings are collected and analyzed, they are used to provide feedback to forecasters, and if an obvious missed forecast has occurred, there may be a case study with regard to the event.
Examples of areas for improvement

Several areas of improvement were noted by key informants:

  • Five of the 29 key informants noted a general desire for more data to better understand users' needs. Restrictions on the conduct of public opinion research and the expense associated with running focus groups has limited this work, but these key informants noted that the SMT SA needs to improve how it obtains client feedback, to better understand if they are meeting client needs. It was noted, however, that efforts are being made to improve how feedback obtained from end-users by the National Inquiry Response Team is shared within the organization.
  • Five key informants noted that the SMT SA is working toward developing a verification system locally and nationally, but would like to see the national verification system developed more vigorously. Two of these key informants noted that there may be more national work carried out in the next six to twelve months and that a new national system and working group should enhance information–sharing among regions.
  • Four key informants noted that the SMT SA is strong in terms of collected data at the output level (e.g., related to issuance of warnings and forecasts), but not as strong in terms of tracking impacts at the outcome level.
  • Three key informants noted that they have not yet seen systematic performance data come forward to the Marine and Ice Program Board, and it was noted that, in spite of the "huge leaps forward" in terms of EC's ability to predict accurately for the marine community, there is not enough data to demonstrate this progress.

A 2011 DFO-led evaluation of the CCG's Icebreaking Program63 noted that CCG management had concerns regarding the level of performance reporting from the CIS. At the time of that evaluation, CCG representatives identified that delayed performance reporting from the CIS limited the extent to which the Icebreaking Program could report to Parliament on the full range of activities and outcomes. However, during this evaluation, CCG representatives noted that the CIS had greatly improved performance reporting in recent months.

4.2.2 Achievement of Outputs and Outcomes

Achievement of Outputs and Outcomes
Evaluation Issue: PerformanceIndicator(s)MethodsRating
5. To what extent have intended outcomes been achieved as a result of the program?

Evidence of / views on achievement of intended outputs and outcomes

Outputs:

  • Percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines
  • Percentage of gale warnings issued at least 24 hours in advance of the event
  • Percentage of ice warnings issued and verified prior to the event
  • Frequency (percentage of times) that trained ice service specialists are provided as per partnership agreements

Outcomes:

  • Increased access by target clients to accurate, timely and user-friendly weather and ice information products and services
  • Increased use of weather and ice information products and services by target clients, to reduce their risks of operating in a hazardous environment
  • Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice
  • Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice

Evidence of / views on factors outside the direct purview of the program that may influence / have influenced the achievement of intended outcomes

Document review

Key informant interviews

Performance data analysis

Achieved

A review of the evidence, including evidence based on activities prior to the formal integration of the weather and ice services under the SMT SA in 2010–11, found that the SMT SA is performing at or near many of its targets for its key outputs, such as the delivery of ice forecasts, charts and marine weather warnings. Furthermore, the SMT SA appears to be achieving three of the four listed outcomes, including demonstrating access to and use of SMT products and services by key clients. The evaluation evidence also pointed to an impact on the reduction of emergency interventions, though the evidence was indirect. The fourth outcome, related to reduced delays for maritime transportation, may not be feasible to measure at present. Given the recent establishment of the SMT SA and its associated outcomes, no historic data exist to measure the degree to which these outcomes have seen changes over time.

Achievement of Outputs

Generally speaking, the SMT SA has been performing at or near the vast majority of established targets for program outputs. More specifically, evidence collected from the SMT SA indicates that forecast bulletins have been meeting their performance targets and that ice charts have been just below the stated target64. Furthermore, the provision of ice service specialists has been consistently meeting the performance target.65

Gale warning lead times, and storm and hurricane–force warnings, are not yet performing at the targeted level; this is partly corroborated by mariners' ratings of the timeliness of marine warnings.66 The POD and FAR for storm and hurricane-force warnings have been performing well, while gale force warnings have been somewhat weaker in their performance relative to the stated target.67 More detail regarding these outputs and results is provided in Table 9.

Table 9: Detailed list of Outputs and Results(text description)
Data collectedResults

Ice information products

  • Percentage of ice forecast bulletins that were delivered on-time by region
  • Percentage of ice charts that were delivered on-time by region
  • With ice forecast bulletins delivered on time 96% of the time, SMT is meeting its performance target, as set out in the 2010–2011 PMF, of 100% of products transmitted within 30 minutes, 95% of the time.
  • With ice charts delivered on time 92% of the time, SMT is very close to but not yet meeting its target, as set out in the 2010–2011 PMF, of 100% of products transmitted within 30 minutes, 95% of the time. However, the PMF2010–2011, which set out the target of 95%, set to achieve this target in 3 years.68

Ice information field services

  • Person days delivered as a percentage of person days requested for ice service specialists
  • The person days delivered as a percentage of person days requested averaged 105%, which indicates that ice service specialists are being delivered as per partnership agreements, and thus meeting the 2010–2011 PMF goal of trained ice service specialists being provided 100% of the time.69

Marine weather information products

  • Percentage of gale, storm, and hurricane–force warnings with lead times of 24 hours or more
  • With lead times of 24 hours for marine gale warnings approximately 50% of the time, SMT is not meeting its performance target, as set out in the 2010–2011 PMF, of 80%. However, a date for achievement of this target had not been established in the PMF. In addition, documentation indicates that the existing PMF target was to be revisited to ensure that the targets are realistic.70 No targets had been set for storm or hurricane-force winds, but their lead times were even lower, at 35% and 10%, respectively.71
  • POD and FAR for gale, storm, and hurricane-force warnings
  • Gale warnings are consistently meeting the POD target of .6 or greater, and are meeting the FAR target of .3 or less in the majority of the quarterly periods. Storm warnings are performing similarly to gale warnings. Though there are data on the performance of the hurricane–force warnings, given the rarity of hurricanes the data were not considered sufficiently robust to report as evidence.72
Achievement of Outcomes

At this point, although there was ample quantitative and qualitative evidence to examine each of the proposed indicators, given the lack of historic reporting for the SA the evaluation team was unable to identify changes over time in the indicators that could quantify actual increases.

Outcome 1: Increased access by target clients (OGDs and end-users, both domestic and international) to accurate, timely and user–friendly weather and ice information products and services

There is a range of evidence suggesting that meteorological, oceanographic and ice information products and services are being accessed by the targeted users. Survey data collected by the MSC indicated that a large majority of mariners use marine weather information, and almost half frequently change their plans based on this information, while two thirds of mariners indicated that they usually followed the recommended ice routes. Mariners are very satisfied with the timeliness, accessibility / user-friendliness and usefulness, but somewhat less satisfied with the accuracy, of marine products. A large majority of participants found weather information products and ice products to be important and that the warning system was meeting their needs.73

Another indicator of access is the number of website hits. In recent years, there have been, on average, over 30 million visits annually to the marine weather portion of the Weatheroffice website and over 1.5 million visits on the Really Simple Syndication feed for marine products.74 Furthermore, surveys indicate that the EC Weatheroffice website is one of the most frequently used sources of marine weather information by mariners. These surveys also indicated that the CCG Continuous Marine Broadcast was another frequently used source of marine weather information.75,76 The CCG obtains its information for the broadcast from EC.

Figure 1 presents the distribution of marine weather sources, based on the survey of end-users. It shows both the general sources of maritime weather information (i.e., all sources accessed by end-users) and the main source of maritime weather information (i.e., the primary source accessed by end-users). These survey findings support the results on access noted above, as 27 of the 33 respondents (82%) indicated that the EC Weatheroffice websiteis one of their sources for marine weather information, and 19 (58%) indicated that the site is their main source of weather information. After the Weatheroffice website, sources such as other Internet sites, Weatheradio, the continuous marine band frequency, the marine radio frequency, radio, television and the CCG's communications system were all highly cited as sources of weather information. The Weatheroffice website, however, was the most–cited mainsource of information.

Figure 1: Distribution of Marine Weather Sources Used by Respondents (text description)

Figure 1: Distribution of Marine Weather Sources Used by Respondents

Feedback received from attendees at two marine events attended by EC representatives identified an access issue with Weatheradio broadcasts: attendees remarked that because these broadcasts alternate between separate English and French broadcasts at fixed intervals, longer broadcasts are frequently cut off before completion, as the broadcast must switch to languages at the scheduled time, thus posing a safety risk and limiting the utility of the product for mariners.77,78 The issue of the alternating broadcast was also highlighted by about half of the key informants interviewed (including individuals external to government), by one respondent in the survey of end-users, and through a review of program documents.79

Outcome 2: Increased use of weather and ice information products and services by target clients (OGDs and end-users, both domestic and international) to reduce their risks of operating in a hazardous environment

As with access to marine forecast products, the evaluation team found a range of evidence to suggest that SMT SA products and services are being widely used by the target populations. The majority of key informants (12 of the 18 that provided a numeric ranking) indicated that the program has increased the use of SMT products by end-users to a significant degree. For example, key informants noted the following:

  • An increased availability of satellite data has allowed the SMTSA to broaden and customize the forecast products it can supply to users, and allowed private industry providers to develop products for targeted populations (e.g., oil rigs), though there is not yet sufficient evidence to conclude whether this has led to increased usage among targeted populations.
  • There is increasing interest in forecast information and products at marine boating shows.
  • Many TV stations serving maritime populations now read weather warnings.

As noted in Figure 1, evidence from the survey of end-users also suggested that SMT SA products are used by targeted stakeholders. The ECWeatheroffice website, radio broadcasts, Weatheradio, and television were frequently identified as SMTSA products used by end-users, and the EC Weatheroffice website was identified as the main source of weather information by a large majority.

As noted in Figure 2, respondents to the survey of end-users also provided high ratings (using a five–point scale) on the accessibility, usefulness, ease of understanding, timeliness and accuracy of the weather information that they used. Respondents generally provided a rating of either 4 or 5 for each item. Usefulness and ease of understanding were most likely to be rated 5 (excellent), by 53% and 50% of respondents, respectively. Respondents provided marginally lower ratings on accuracy than the other items, as shown below, with only 6% rating this item as a 5, although nearly 56% provided a rating of 4.

Figure 2: Rating of the Marine Weather Service (text description)

Figure 2: Rating of the Marine Weather Service

Other sources of evidence, such as the 2004 Marine Weather National Services Survey, also suggest that historically the use of SMTSA products and services has been high. Some findings from the survey:

  • Of the 98% of mariners that use weather forecasts, 98% use marine weather forecasts an average of three times per day.
  • The daily ice charts (86%) and recommended ice route (75%) are important ice information products (rating of four or five on a five–point scale). The thirty-day forecast (35%) and seasonal outlook (34%) are rated as less important.
  • Half of participants changed their plans in response to marine weather information "once in a while," one third "most of the time," and one tenth "almostalways."
  • Two thirds (66%) of participants either "always" (29%) or "most of the time" (37%) followed the recommended ice route when operating in the ice.

Outcome 3: Reduced number of emergency interventions (by emergency response teams and/or CCG) due to marine weather or ice

An unqualified assessment of this outcome was not possible, given the difficulty in establishing direct attribution of emergency interventions to SMT programs and activities. Nevertheless, the evaluation team was able to examine evidence from marine investigations of the Transportation Safety Board and key informant interviews that referred to general trends linking weather and ice information to marine safety. From 2005 to 2009, there were 36 marine investigations published by the Transportation Safety Board. Of these reports, most incidents (75%) tended to occur during favourable weather conditions. Of those involving inclement weather or difficult seas, weather reports were accurate and had given sufficient notice to mariners in all but one case, where an inaccurate report was deemed to have increased the risk of an accident.

Table 10: Transportation Safety Board–Summary Table 2005-2009 (text description)
YearTotal no. reportsNo. incidents where weather/ice conditions were a factorNo. reports where service provided by MSC was accurateSummary of incident
2009711Quartering seas
200870NA 
2007511Quartering seas
2006124380Squall
Quartering seas
Hurricane-force winds (as predicted by U.S. Coast Guard)
Strong winds81
20051533Fog
Caught in ice
Poor weather
Total4698 

Key informants were generally cautious in addressing this outcome indicator directly, and several respondents did not provide a rating on the degree to which the SMT SA contributed to reducing emergency interventions, noting that the role played by EC is important, but by no means the only factor in reducing the number of emergency interventions.

Of those who provided a view regarding the degree, responses ranged from moderate to significant. A number of examples were provided of how SMT products and services have led to reductions in the number of emergency interventions associated with marine weather or ice, including:

  • lobster season no longer has a pre-set opening day--it varies depending on the forecast, to protect the safety of mariners;
  • emergency intervention representatives consult EC much more frequently than previously and make greater use of EC's information for planning; and
  • ice information was used to issue a warning and help in a situation regarding a missing teenager in the north, by enabling the search areas to be better targeted.

Also, all three key informants from OGDs were generally positive regarding the MSC's role in providing accurate weather information to assist and support maritime emergency responses.

Outcome 4: Reduced unexpected delays encountered by commercial marine transports due to marine weather or ice

The majority of respondents do not believe that this outcome can be assessed at this point in time, because there are no data available to address the indicator, and it would be difficult to know how weather and ice information could have potentially affected the advanced planning for travel routes. Examples of reductions in unexpected delays that were nonetheless noted by respondents included: revisions to ferry service schedules in the Pacific and Atlantic regions as a result of forecasted information, and a reduced need for ship navigation through ice-bound waters as a result of regional CCGadjustments to shipping routes; these adjustments were based on detailed information provided by the SMTSA. Key informants noted, however, that the MSC plays a limited role in affecting this area of maritime navigation and that, at the very least, this outcome needs to be considered a shared outcome with other parties.

CCG documentation indicates that only a small minority of service delays for clients in 2008-2009 were due to ice conditions (13%) or weather or tide conditions (4%). Once again, though, this information is not adequate to determine whether these delays were anticipated or unexpected, or what role SMT-related products may have had in mitigating their impact.82

Unintended/unanticipated outcomes

Only one unintended outcome of the SMT SA was identified by multiple key informants. This issue related to unforeseen impacts on service delivery as a result of introducing a standardized national approach to the forecast for the Weatheradio broadcasts. Prior to the initiative, which introduced a standardized two–day forecast on all broadcasts, there had been a three-day forecast included in broadcasts in the Atlantic region, and a one–day forecast on broadcasts in other parts of the country. The modification to this broadcast schedule in the Atlantic region and the resulting frustration among clientele in the region had been unintended.

External factors affecting the achievement of outcomes

A number of issues were identified by key informants as areas outside of the MSC's direct control that have, or may have, impacts on the performance of the SMTSA. Approximately one quarter of the key informants pointed to the risks of relying on other collaborating organizations for funding and service delivery. For example, it was noted that EC is increasingly relying on external organizations for the provision of satellite monitoring data, which have become an important source of meteorological information. Furthermore, SMT SA services could be greatly reduced, at least in the short term, if an existing collaborating organization (CCG being the most frequently cited) were to undergo budget cuts or a change in business model. Another cited example was the fact that EC relies on CCG for transportation when servicing weather buoys, and there have been instances where this has resulted in several months passing without service for a particular buoy, thereby resulting in a loss of monitoring data.

4.2.3 Demonstration of Economy

Demonstration of Economy
Evaluation Issue: PerformanceIndicator(s)MethodsRating
6. Are Services to Marine Transportation achieving their intended outcomes in the most economical manner?
  • Views on whether the resources and inputs into the program are managed economically
  • Clearly defined and understood governance structure, including program processes, roles, responsibilities and accountabilities
  • Views on whether roles, responsibilities and accountabilities are functioning as intended
  • Views on the appropriateness of program activities, processes, and governance structures
  • Document review
  • Financial analysis
  • Literature review / international comparison
  • Key informant interviews
Progress Made; Attention Needed

Generally, the SMT SA appears to be managing its resource inputs in an economical way, including reallocating resources from existing areas to address new initiatives, as was the case with METAREAs Initiative funding for 2010-11. Additionally, effective governance and priority-setting mechanisms have been established. There are, however, some areas where further clarification of roles and responsibilities may be beneficial, particularly regarding certain responsibilities between ECand DFOrelated to marine weather issues for buoy maintenance and storm surges.

Resource Allocation

The numbers in Table 11 represent expenditures for the SMT SA for the last four fiscal years. The budgets at the start of each fiscal year were not available from the Department, thus precluding an analysis of budgeted versus actual expenditures; however, the Evaluation Division was informed by the Financial Services Division that this SA tends to manage its budget very closely and that actual expenditures essentially equate to the total budget for the program. As noted previously, the METAREAs Initiative is outside the scope of the evaluation as it is too new to evaluate, but expenditures for METAREAshave been included in Table 11, in order to provide a complete picture for the 2.2.2 PAA element (the SMT SA).

Table 11: 2007-2009 Annual Financial Expenditures for 2.2.2 Services to Marine Transportation83 (text description)
Financial Expenditure2007-082008-09
MarineIceTotalMarineIceTotal
FTEs319512632108140
Salary2,3862,8615,2472,6392,7715,410
O&M1842,1192,3022122,6342,846
Capital077770337337
VNR-Salary13,8073,80805,2005,200
VNR-O&M05,7165,71605,1195,119
TOTAL*2,57014,57917,1502,85116,06218,912

 

Table 11b: 2009-2011 Annual Financial Expenditures for 2.2.2 Services to Marine Transportation83 (text description)
Financial Expenditure2009-102010-11
MarineIceTotalMarineIceMet-
areas
Total
FTEs32108140321047144
Salary2,8563,0375,8933,0932,5886566,336
O&M1241,5141,6381141,8692352,218
Capital01161160355177532
VNR-Salary55,5885,59405,26105,261
VNR-O&M05,1165,11624,13824,142
TOTAL*2,98515,37218,3573,20914,2121,06918,491

* Numbers are shown in $000s (except FTEs).

Of note, approximately two thirds of the budget for ice services comprises VNR funding, which primarily represents fees for services to CCG. Although the partnership with CCGbrings many benefits, this reliance does present an ongoing risk to the SMTSA, as its ability to deliver on mandated services is highly dependent on this continued funding. Furthermore, it poses a further challenge for long-term expenditure planning, as SA program management must ensure flexibility to adapt to possible changes in funding levels due to CCGbudget variations. Additionally, the SMTSA, and especially the marine weather component, is reliant on the MSC's Weather, Observations, Forecasts and Warnings SA (PAA element 2.1.1), as work conducted by meteorologists and other MSC specialists to support services to marine transportation is built upon the "core" services resourced under this SA. The SMT SA could not perform its work at current resource levels without leveraging the work done by this other SA.

Also of note, although FTEs did increase moderately in 2010–11 with the introduction of the new METAREAs component of the program (additional 4 FTEs, representing less than a 3% increase), it did not come with an overall budget increase for the PAA element. Rather, expenditures for the Ice Research and Ice Warnings and Forecast Information components of the program were reduced by essentially the same amount as the funding now identified for the METAREAs Initiative. Given that the METAREAs Initiative is in early implementation, it is too early to assess its impact on the achievement of results.

As demonstrated by the numbers in Table 11, and portrayed in Table 12, the SA had an increase in expenditures of approximately 10% between fiscal years 2007–08 and 2008–09, for both the marine and ice components of the SA. For the remaining two years of the study period, expenditures for the SAremained relatively stable year over year, although there was a slight fluctuation between the two components.

Table 12: Year over Year Changes in Expenditures(text description)
Expenditure2008–09 vs. 2007–082009–10 vs. 2008–092010–11 vs.
2009–10
MarineIceTotalMarineIceTotalMarineIce**Total
Year over year changes in spending11%10%10%5%-4%-3%8%-1%1%

 ** Includes Ice Research, Ice Warnings, Forecasts & Information and METAREAs.

Table 13 indicates the percentage distribution of resources across the three branches that receive financial resources under the SMT SA. There has been little change in the distribution of financial resources, either to the program components or across the three branches, since 2007–08. There were no comments from key informants suggesting that this distribution was likely to change, barring concerns regarding a loss of support from the key partner, i.e., the CCG. The low degree of variation across the four years appears to suggest that there does not appear to be a need to adjust the SMT SA's overall operational approach.

Table 13: Percentage distribution of resources over last four fiscal years84 (text description)
Distribution2007-082008-092009-102010-11
MSCS&TCIOBMSCS&TCIOBMSCS&TCIOBMSCS&TCIOB
Marine15%0%0%15%0%0%16%0%0%17%0%0%
Ice69%1%15%69%1%16%68%1%15%66%1%16%
% of Yearly Total84%1%15%84%1%16%84%1%15%84%1%16%
Governance and Priority Setting

Though generally positive, there were a variety of responses regarding the extent to which the SMT SA governance and priority–setting mechanisms were supporting the effective management of resources. Since June 2010, the main governance mechanism for the SA has been the Marine Weather and Ice Program Management Board. About one quarter of the key informants were positive in their appraisal of the SMT SA's governance and priority-setting structure, noting that the new board has been effective in setting priorities, such as approaches to issuing marine forecasts and watches. The remaining key informants indicated that the newness of the board limited the extent to which it was either willing or able to comment on its functioning, although there was a general sense of optimism that the new board structure would be effective in managing SMTSA programs and services.

EC key informants were also asked to comment on the extent to which the two SMT SA components (weather and ice) could share resources and expertise in order to minimize costs and increase effectiveness. It was generally noted that the training required for a weather meteorologist is significantly different than for an ice specialist and, aside from co-locating the two functions, other efficiencies, such as sharing or transferring staff from one area to another, would be impractical.

Several key informants noted, however, that there is some duplication in terms of weather analysis as it relates to wind forecasts. Ice specialists use wind speed direction to track ice flow direction and, as the CIS uses remote sensing to track ice and wind, there are opportunities for both the weather and ice components to share these data. Others also noted that there is room for more integration, as there are natural reasons why ice and marine weather information should be more intertwined (for example, ice changes shoreline impacts and storm surges). Further, end-users do not separate the two functions, so opportunities may exist for additional coordination of information dissemination, warnings, websites, modelling and electronic charting.

It was noted, however, that there are expectations that the METAREAs Initiative will contribute to finding efficiencies in terms of how the marine forecast and ice components work together, and to finding opportunities to work with EC's S&T community toward developing further services unique to northern communities. Given the early implementation of this initiative, however, it was not possible to discuss examples of how the initiative has yielded these efficiencies.

Clarity of Roles and Responsibilities

The clarity of roles and responsibilities among federal government stakeholders involved in the delivery of services to marine transportation was assessed. Key informants generally pointed to a clear articulation of roles and responsibilities. Several respondents noted, however, that there is room for improvement in this area, specifically the need for clarity regarding the formality of shared responsibilities and costs between EC and its collaborators. The federal government's ice services program is delivered through a formal MOU between EC (specifically the MSC) and DFO (specifically the CCG),85 which generally helps define roles. It was noted by EC and DFO respondents that the CIS relationship with the CCG is generally working very well: both parties understand what each is expected to deliver, due to the existing 5–year MOU between the two parties.

However, key informants noted that there were still too many informal arrangements in place between SMT and its collaborators, and identified a lack of clarity in some areas, particularly for the marine weather component where the definition of roles and relationships is much more informal. The most frequently identified areas in need of increased clarity were regarding responsibilities for deployment and maintenance of weather buoys and storm surge warnings:

  • Deploying and servicing buoys – Key informants noted that the CCG has discussed charging EC to deploy weather buoys. Key informants at both EC and DFO used the term "ahandshake agreement" to define the current relationship between the two departments regarding the displacement of buoys. Although this relationship has historically been effective, if resources become increasingly stretched this informal agreement may no longer be feasible and EC could be faced with an additional, unexpected cost to deliver on SMT services. There is currently no national MOU for servicing buoys, though one does exist for Pacific and Yukon Region, which key informants noted was primarily developed as a result of existing relationships at the managerial level. On a related note, key informants also noted that there is some confusion as to who is responsible for the database of buoy information for record keeping and historical analysis.
  • Storm surge – The storm surge warnings issue arises from the fact that delivery of this capability is not consistent across all regions and is not clearly defined. Currently, EC provides the publicly accessible data86 and warnings to the east coast and Great Lakes, while DFO provides this information to the west coast and Quebec. A key difference noted by key informants in relation to this responsibility was that the MSC is an operational organization maintaining an operational staff 24/7, whereas DFOactivities tend to be conducted by scientists solely during regular business hours.

Another area where clarity could be improved, to ensure that program costs are minimized, is contingency plans for emergency situations (for example, oil spills); these plans are not as clear as many (including external stakeholders) would like. Given the potential costs associated with a lack of clarity over emergencies, there were calls to have roles clearly laid out between all key players (e.g., EC, DFO, TC and DND) when an incident occurs in Canadian waters. A need was also identified for better clarity over responsibilities in the Arctic, particularly as the costs associated with working in the North are higher than in southern Canada. A noted example was the need to find the most cost-effective solutions for deploying weather buoys and maintaining a minimum level of verification through observations, although, again, it was identified that the METAREAs Initiative will likely address this.

4.2.4 Demonstration of Efficiency

Demonstration of Efficiency
Evaluation Issue: PerformanceIndicator(s)MethodsRating

7. Is the Services to Marine Transportation Program undertaking activities and delivering products in the most efficient manner?

  • How could the efficiency of the program's activities be improved?
  • Are there alternative, more efficient ways of delivering the program?
  • Views on whether good value is being obtained with respect to the use of public funds
  • Views on how the efficiency of program activities could be improved
  • Views on whether there are alternative, more efficient ways of delivering program activities and outputs
  • Document review
  • Financial analysis
  • Literature review / international comparison
  • Key informant interviews
Achieved

The evidence suggests that the program is generally delivering its services in an appropriate and efficient manner and has taken steps to improve efficiency. There may be opportunities for further efficiencies associated with new technologies for dissemination, although this must be carefully managed so that mariners who rely on older technologies continue to receive the information they require for safe and efficient operation.

Overall Views of Efficiency of SMT SA Products and Services

Generally, key informants expressed a positive view regarding the efficiency of the SMT SA, and noted several examples:

  • The majority of key informants, including program representatives and end–users, expressed views on the value of SMTSA products and services with respect to the use of public funds, by noting that the quality and production of forecasts and warnings are high and provided in a format and language that is understood by the marine community, and that the delivery of services is appropriate and efficient given available resources and the need to provide services 24/7. This is consistent with findings from a recent evaluation which found that EC's weather prediction service is generally viewed as providing high-quality information and products.87
  • It was noted by EC and CCG key informants that the CIS has been receiving decreasing funding from the CCG to maintain the same level of service (although funding was stable for the four–year period of the evaluation, there has been a decrease of approximately $7 million since the MOU in 1995). Evidence from the key informant interviews and document review suggests that the main reason that the CIS has maintained the same level of service despite budget cuts has been cost savings associated with the increased use of radar and remote sensing technology and the decrease in the use of fixed-wing aircraft observation.
  • Two key informants also noted that international collaboration has been effective, particularly in the Great Lakes region, although more could be done with the United States in developing common products and dissemination mechanisms. No international stakeholder was interviewed for this evaluation, although a senior official in the U.S. federal government was interviewed recently in a related evaluation,88 commenting that CIS ice observation products have improved dramatically, primarily because they have much better sources of information now that they use radar and remote sensing technology that has revolutionized the entire field. Furthermore, this official noted that Canada and its U.S. counterparts have solidified their relationship and that the organizations are undertaking collaborative projects and expanding joint activities.
  • Dissemination systems were noted as working reasonably well given the variety of dissemination tools offered. Two key informants expressed the view that the dissemination methods for the marine forecasts were meeting needs for this targeted group of users better than the dissemination methods available for broader public offering.
  • Consultations with other government agencies (e.g., Canadian Marine Advisory Council) and communication at trade shows have been an effective, low-cost approach to improving the reach of SMTSA products and services, and to providing staff with the opportunity to engage with the maritime community.

Key informants also identified several areas that can potentially improve efficiency:

  • The need for a broad range of dissemination systems was the most frequently noted. It was noted that the SMT SA is obliged to deliver its information in a wide variety of systems to meet the diverse needs and technological capacities of its end–users. For example, radio is still considered a key source of dissemination for mariners well out of port, though there are limitations due to expense and reach. Meanwhile, the Weatheroffice website was the most highly used source of marine weather information among external users surveyed, and several program representatives believed there were opportunities to explore more technologically advanced capabilities such as opportunities to leverage mobile phones. Although no solution was provided, many believed that improvements to the dissemination system would improve the efficiency of SMT SA services, through leveraging new technologies.
  • Equally cited by key informants was the topic of public-private partnerships. A growing number of firms obtain publicly accessible data from EC sources to produce very specific and/or localized forecast information for specific clientele, such as providing information on the safety limits of the ice edge to remote northern communities (the evaluation team found six examples of such organizations and spoke to representatives of two such groups). EC and industry representatives commented during the open-ended interviews that the provision of specialized information would push EC's existing resources too far, and that private industry was in a better position to meet the demands of specific clientele. Although one key informant suggested that EC could also raise revenue by charging a fee-for-service to deliver similar localized information, the majority of respondents who commented on this issue suggested that the current approach, in which EC manages the general network of national data and private providers use this data to address specialized needs in the market, appears to be effective and is more in line with open government policies in Canada and the United States.
  • Five key informants indicated that the SMT SA needs to find approaches to improve verification and observation networks because data availability is sparse, particularly in the North.

Key informants did offer several alternative approaches, however, to improve efficiency and maximize the use of current resources. The majority of alternatives were based on expansion or better use of technology, which is perhaps not surprising given the cost savings and improved efficiency for SMT SA services based on the increased use of technological innovations such as satellite-based data.

Approximately one quarter of the key informants indicated that the SMTSA should explore the better use of technology for dissemination, such as applications for smart phones or alerts via text messages to cellphones. Though there will be technical challenges to making use of enhanced dissemination products, key informants indicated that there is a need to look at the availability and feasibility of new technologies and that both SMTSA components need to take better advantage of technological advances and innovations. However, as was noted above, given the wide range of end-users, many mariners still use and are attached to older sources of technology, making it difficult and potentially harmful for the SMT SA to replace one dissemination tool with another.

Comparison of Costs

As a final source of evidence related to the efficiency of SMTSA products and services, the evaluation team examined an external benchmarking study of the meteorological service organizations in seven countries, including Canada's MSC. Although the study is not specific to marine and ice information, it contained an international cost comparison for the provision of meteorological services among the organizations shown in Figure 3.89 The figure shows relative spending, not only in terms of the cost per capita, but also the cost per square kilometre.

Figure 3: Relative Spending on Meteorological Services90 (text description)

Figure 3: Relative Spending on Meteorological Services

The United States and Finland are of particular interest to the SMTevaluation, as those countries' meteorological service organizations also have ice services. Regarding Finland, its Finnish­ Meteorological Institute (FMI) experienced budget increases from 2007–2009, mainly due to increases in external funding and extra government revenue to increase services to the marine sector and research for the public good. The FMI provides many of the same services to marine transportation as EC's MSC, including providing forecasting models to its coast guard, among other clients.91Further, the FMI website contains information on marine weather and the Baltic Sea, including weather forecasts for shipping, sea level, ice conditions, wave forecasts, and sea temperature forecasts.92

Figure 3 indicates that Canada spends approximately twice as much per person as does the United States and approximately half as much as does Finland. In terms of spending per square km, Canada spends less than both the United States and Finland.


21. EC and DFO. 2005. IISPA.

22. CCG. 2009. CanadianCoast Guard Fleet Annual Report – 2007-2008, p. 12.

23. These issues are discussed in detail in section 4.1.3.

24. United Nations. 1982. United Nations Convention on the Law of the Sea.

25. IMO, 1974 and later amendments. International Convention for the Safety of Life at Sea.

26. Article 4 and 5 of the Department of the Environment Act, as quoted in WES Consultation Document August 28 2008.

27. WES Consultation Document August 28 2008.

28. Standing Senate Committee on Fisheries and Oceans. 2009. Rising to the Arctic Challenge: Report on the Canadian Coast Guard, pp. 21-27, 42-45.

29. Government of Canada. 2010. The Budget Speech 2010: Leading the Way on Jobs and Growth.

30. Government of Canada. 2010. Speech from the Throne. Delivered March 3, 2010.

31. Government of Canada. 2007. Speech from the Throne. Delivered January 26, 2009.

32. Finance Canada. March 4, 2010. The Budget Speech 2010: Leading the Way on Jobs and Growth, pp. 106, 68, 111, 255, 231.

33. Terms of Reference - MIPMB v 20100608.

34. As noted previously, the SMT PAA element is the entity being focused on in this evaluation.

35. EC. 2006. Environment Canada's Sustainable Development Strategy 2007-2009, p. 4.

36. Office of the Auditor General of Canada. 2008. 2008 December Report of the Commissioner of the Environment and Sustainable Development, Chapter 2--Managing Severe Weather Warnings--Environment Canada.

37. EC and DFO. 2005. IISPA, p. 5.

38. In 2008, EC obtained certification for its weather and environmental services function under ISO 9001 following implementation of its QMS.

39. EC. 2010. 2010-2011 Performance Measurement Framework Weather and Environmental Services.

40. EC and DFO. 2005. IISPA. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

41. EC. 2008. Weather and Environmental Services QMS Manual.

42. EC. 2010. Meteorological Service of Canada: Marine Forecast Verification Project – Project Charter.

43. A draft PMF was under development during the evaluation and was shared with EC evaluation team members.

44. EC. 2010-11. Performance Measurement Framework – Weather and Environmental Services.

45. The 2010-11 draft PMF outlined the following outputs and outcomes: percentage of ice charts and forecasts transmitted within 30 minutes of prescribed deadlines; lead times of marine gale warnings of 24 hours or greater, in 80% of events; and level of satisfaction of the marine communities with respect to weather, wave and ice information received by EC. Other outputs and outcomes were developed from PAA element descriptions, evaluation scoping interviews, and consultation with the Departmental Evaluation Committee.

46. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

47. Earlier wording in reports: "the number of duty days delivered versus requested," which is consistent with wording in IISPA: "# duty days delivered vs. requested (%)."

48. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

49. EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

50. Defined as the number of correct warnings as a percentage of the total number of events.

51. Defined as the percentage of warnings that are incorrect.

52. EC. Maritimes Verification and Newfoundland Verification, 2007-2010.

53. EC. 2004. Marine Weather Services National Survey. Final Report.

54. Recreational mariners represent a minority of the total mariner population and are not the primary focus of the SMTmandate.

55. DFO. 2009. Icebreaking Program Report on Performance: Winter Operations 2008-2009.

56. EC. 2011. Weatheroffice website statistics.

57. Transportation Safety Board. 2011. "Marine Statistics."

58. DFO. 2009. Icebreaking Program Report on Performance: Winter Operations 2008-2009.

59. EC. 2004. Marine Weather Services National Survey. Final Report.

60. DFO. 2008. Levels of Service Review: Summary of Client Comments. Draft Report.

61 EC. 2010. Canadian Marine Advisory Council Maritimes Region Marine Day.

62. EC. 2011. North Atlantic Fish & Workboat Show 2010 November 26-27, 2010.

63. DFO. April 2011. Evaluation of the Icebreaking Service.

64. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

65. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

66. EC. 2004. Marine Weather Services National Survey. Final Report.

67 EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

68. EC. CIS Monthly Performance Measurement Reports, July 2007 to August 2010.

69. EC. CIS Monthly Performance Measurement Reports, July 2007 to Aug 2010.

70. EC. 2010. Marine and Ice Program Management Minutes Sept. 24 2010.

71. EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

72. EC. Public and Marine Warning Performance Report, Jan.-Mar. 2010 and Oct.-Dec. 2009.

73. EC. 2004. Marine Weather Services National Survey. Final Report.

74. EC. 2011. Weatheroffice website statistics.

75. EC. 2004. Marine Weather Services National Survey. Final Report.

76. EC. 2010. Canadian Marine Advisory Council Maritimes Region Marine Day.

77. EC. 2010. Canadian Marine Advisory Council Maritimes Region Marine Day.

78. EC. 2011. North Atlantic Fish & Workboat Show 2010 November 26-27, 2010.

79. EC. 2011. North Atlantic Fish & Workboat Show 2010 November 26-27, 2010.

80. Includes one report where data were provided by the U.S. Coast Guard.

81. As noted in the description, unexpected strong winds contributed to the single maritime accident in which weather information was deemed to be inaccurate.

82. DFO. 2009. Icebreaking Program Report on Performance: Winter Operations 2008-2009.

83. The figures shown as "Marine" represent all line items that map to Marine Weather Warnings Forecasts and Information in the departmental financial coding system, and the figures shown as "Ice" represent the values for both Ice Research and for Ice Warnings, Forecasts and Information.

84. Source: Financial Management Branch, EC. May 2011.

85. EC and DFO. 2005. IISPA.

86. Storm Surge

87. EC. 2011. Evaluation of Support of Weather Prediction.

88. EC. 2011. Evaluation of Support of Weather Prediction.

89. Kerry A., Wilkinson C., van Mossel J. June 4, 2009. Benchmarking Study on Business Models for the MSC. Final Report. p. 21.

90. Ibid., p. 21. Note: amounts are presented in U.S. dollars.

91. Ibid., p. 9.

92. Wind and pressure forecast for Baltic Sea

Date modified: