This page has been archived on the Web
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
Evaluation of the Services to Marine Transportation Sub-activity
This report presents the results of the Evaluation of Services to Marine Transportation (SMT) conducted by Environment Canada’s (EC’s) Audit and Evaluation Branch between September 2010 and May 2011. It addresses the activities that fall under the Department’s Program Activity Architecture (PAA) 2.2.2 sub-activity (SA).
Although the evaluation was conducted within the context of a joint review and evaluation, this report contains the findings of the evaluation component only, and addresses the standard evaluation issues pertaining to program relevance and performance. The period examined is from 2007-08 to 2010-11.
EC’s SMT SA provides marine users with information to support decision making for marine activities and warnings for weather-related threats to safety. It serves the needs of maritime communities, and supports Canada’s obligations in providing safety information for international marine traffic. EC’s SMT SA comprises two main activities: the provision of marine weather information and the provision of ice information.
Within EC, delivery of the SMT SA is performed by the Meteorological Service of Canada (MSC), the organizational branch with responsibility for producing and disseminating weather and environmental information, including that related to ice.1 The MSC also relies on the Atmospheric Science and Technology Directorate of the Science and Technology Branch (S&T Branch) as a key contributor to meteorological research and development, and the Chief Information Officer Branch for activities related to information management and information technology. The activities performed under the SMT SA also rely heavily on the broader base of weather prediction services provided by other groups within the MSC and the S&T Branch, as delivered under the Department’s PAA activity 2.1 – Weather and Environmental Services for Canadians.
The evaluation methodology included a review of relevant federal and departmental documents, as well as key informant interviews with program stakeholders and a questionnaire that was distributed to end-users.
Findings and Conclusions
The following represents a summary of the overall findings and conclusions of the evaluation.
Multiple sources indicated that there is a continued need for ice and marine weather information activities and products, as these activities help Canadians reduce risks posed by changing weather conditions.
Findings from multiple sources indicated that the SMT SA is aligned with the legislated mandates of the involved departments, and with responsibilities assigned through agreements and other mechanisms for collaboration. Furthermore, the evidence indicates that the MSC is obligated by various acts and agreements to deliver SMT SA services, including in the context of ongoing northern development.
For the most part, evidence found that EC’s SMT SA services complement services in other government departments. However, concerns were expressed by EC and collaborating department representatives regarding the extent to which some areas of shared responsibility are clearly articulated. There is additional concern that the Department’s ability to deliver on its responsibilities is vulnerable, as these services are reliant on key partners or collaborators.
In addition, the SMT SA is aligned with and supports federal government priorities related to health and safety, the economy, infrastructure and sovereignty, as well as departmental strategic outcomes and priorities related to ensuring that Canadians are equipped to make informed decisions on changing weather, water and climate conditions.
Key informants noted that one area of expanding interest for the federal government, and an area in which the MSC will likely provide increased marine weather and ice services, is northern development. The MSC has been resourced to undertake new approaches to providing services to Arctic communities and industries under a newly established Meteorological Areas (METAREAs) Initiative that is funded under this SA. However, as this initiative had only begun at the outset of the evaluation, it was not included in the analysis and will be evaluated by EC at a later date.
The evaluation found evidence of a strong commitment to performance measurement, supported by the MSC’s ISO 9001 Quality Management System. Considerable data collection is occurring, and program management appears to recognize the value of performance measurement.
Although the activities that comprise the SMT have existed within the MSC since its origins 140 years ago, the grouping of the provision of marine weather information and ice information as one PAA element, known as the SMT, was just recently introduced with the 2010-11 PAA. As such, the existence of a Performance Measurement Strategy that presents a logic model for the PAA element and a cohesive Performance Measurement Framework (PMF) for the entire element does not yet exist, although efforts are being made in this area.
Nevertheless, the evaluation found that the SMT SA was able to provide data and targets on key outputs that were identified for the purpose of the evaluation, such as the delivery of marine weather warnings and ice forecasts and charts, and that, furthermore, it is performing at or near the vast majority of these targets. With regards to the availability of performance measurement related to outcomes, data were also available for the outcomes identified for the purpose of the evaluation, although baseline data that would support an understanding of how performance has changed over time was limited.
The evaluation found evidence of strong performance regarding outcomes related to usefulness and ease of understanding of weather and ice information products. There is a very high incidence of mariners accessing and making use of the marine weather and ice information provided by EC, and overall satisfaction with the products and services was high. Marine and ice information is disseminated to mariners with a broad range of methodologies. Although the EC Weatheroffice website was identified as the most frequently used source of marine weather information, evidence found that most mariners relied on more than one source or technology for their information needs, and further, that users of marine information are a diverse group who include both those who rely on receiving information in the same manner in which it has been provided for many years, as well as those who would like to see further enhancements that leverage advanced technological capabilities. This presents a challenge for EC from an efficiency perspective, as it makes it very difficult to find cost savings to fund new advancements from the removal of historic products.
The SMT SA operates on a budget of approximately $18.5 million per year, and spending on these activities has been relatively stable over the four years of the study period. EC’s SMT is not, however, a self-contained function, but rather is able to deliver on its outcomes by relying both on the broader underlying capacity of the MSC for weather and environmental services as well as its important relationships with its key partner, the Canadian Coast Guard (CCG), and many other collaborators and stakeholders, including Transport Canada, the Canadian Space Agency, the World Meteorological Organization, and the U.S. National Ice Center, among others. These relationships and the cooperative, collaborative manner in which the Department operates in this area, supports EC’s ability to deliver marine transportation services in a cost-effective manner. The relationship with the CCG appears strong, for the most part, although there are areas where roles and responsibilities could be clarified. In particular, clarifying roles related to servicing weather buoys and providing storm surge warnings were identified as areas lacking clarity. Addressing areas of ambiguity will have the added advantage of ensuring that the inherent risk associated with reliance on another party is minimized.
The following recommendations are based on the findings and conclusions of the evaluation. The evaluation recommendations are directed to the Assistant Deputy Minister (ADM), MSC, in light of the ADM’s responsibility for the overall management of SMT activities.
During the conduct of this evaluation, the evaluation team was able to rely on a variety of performance data from the SMT SA’s initial draft 2010-11 PMF to develop a preliminary assessment of various aspects of the SMT SA’s performance. Working with this, the evaluation team developed a set of outputs and outcomes in order to support a preliminary assessment of performance. As noted in the report, there was some concern expressed by SMT management at the onset of the evaluation that the outputs and outcomes used for the evaluation may not be adequately aligned with SMT services and activities, and that management may not be able to collect the required data. It is notable that the program was able to produce data across these outputs and outcomes, and this does represent a significant step toward ongoing performance reporting. Additionally, it is clear that the SA recognizes the importance of formalized performance measurement, through ongoing development of the SMT SA’s PMF. However, several gaps in performance reporting still remain, including a more thoroughly validated set of performance indicators across expected outputs and outcomes, and a reporting strategy that integrates these data sources into a common reporting tool. It is therefore recommended that the ADM MSC, using the evidence presented in this report as a baseline for performance reporting, continue to develop the PMF, including a logic model, expected program outputs and outcomes, and a strategy to collect and report on this information. Attention should be given to the performance reporting requirements of organizations such as the CCG.
The services provided by SMT are mandated by a range of obligations set out in both international and domestic acts and agreements. Furthermore, these acts and agreements link SMT to a variety of federal and international collaborating organizations, and have clearly formalized several key areas of SMT service provision (such as the deployment of ice specialists onboard CCG ships). Although obligations clearly establish the need for SMT to provide maritime weather and ice services, there are nevertheless potential risks to SMT if there are changes in the level or type of service provided, or in the financial and human resource commitments by key collaborating departments. During this evaluation, there was particular concern expressed regarding clarifying the relationship with the Department of Fisheries and Oceans (DFO) / CCG in areas such as the maintenance and servicing of buoys, and the provision of storm surge warnings in various regions. It is therefore recommended that the ADM MSC develop a strategy to identify and address risks posed to the SMT SA due to any areas of uncertainty in its relationships with collaborating departments. As a starting point, this strategy should include an approach to formalize the relationship with DFO/CCG regarding servicing weather buoys and providing storm surge warnings.
It was frequently noted that the SMT SA must use an increasingly wide range of communications technologies to disseminate meteorological information across all potential users, and that program management is limited in the number of media that can be reduced or removed, as it may pose safety risks to end-users that rely on these specific forms of media. Given the growth in available potential methods of dissemination in relation to current resources, it is therefore recommended that the ADM MSC establish a strategy to manage the evolution of dissemination strategies for marine weather and ice information, and to examine the consequences of reducing or removing current dissemination mechanisms. This strategy should be based, to the extent possible, on the documented needs of SMT’s maritime clients and should include cost estimates for each proposed new approach. A process should be established to ensure that the strategy is reviewed and revised on a regular basis.
The MSC agrees with the recommendation.
The MSC already has a PMF in place for the SMT program (PAA sub-activity 2.2.2), and has been updating the program outputs and outcomes as well as strategies to collect and report on this information, for several years. Updates to the expected outcomes and outputs for the 2012-13 reporting cycle are awaiting EC management approval, and are being submitted to Treasury Board Secretariat in the summer of 2011.
As part of the development of a performance strategy for the METAREAs marine services (a new initiative that falls under the SMT program), a logic model and PMF were developed during 2010-11. Through this exercise for the METAREAs, there is transferable information that will be used in the development of a logic model for the SMT program. The MSC is committed to delivering on a logic model and updating its Performance Measurement Strategy by March 2013, after sufficient consultation with its clients and partners.
In the initial IISPA (Ice Information Services Partnership Agreement) between the MSC’s Canadian Ice Service and the CCG, a logic model and performance measures were developed. Reporting on all of them became challenging for both, and thus, in the re-negotiated IISPA, the number and type of performance measures have been reduced and focused significantly, with the intention to refine them in the coming year.
Pre- and post-season meetings with the CCG and clients result in requests for new products, and reductions and/or changes in other products. These are usually captured in the minutes of these meetings and actioned by either party.
The MSC agrees with the recommendation.
The MSC enjoys benefits and synergies from its collaborative arrangements, but also acknowledges the risk posed by uncertainties with respect to its relationships with some collaborating parties, including other government departments. This risk has been identified in the Weather and Environmental Services (WES) Risk Management Profile, although the risk statement does not specifically note the weather buoy and storm surge warning elements associated with the relationship with DFO/CCG. This dimension of the collaboration risk will be more fully reflected in the next update to the Risk Management Profile. The MSC’s Marine and Ice Program Management Board will be tasked with developing strategies to analyze and mitigate these risks, and to present a strategy to MSC senior management before the end of 2011-12.
The MSC agrees with the recommendation.
There is a risk, when in the business of information dissemination, of chasing the newest technologies (e.g., social networking) without fully understanding client needs and the effectiveness of the technologies in meeting these needs. The MSC’s dissemination strategies take into consideration a number of factors, including client needs, technological opportunities, synergies among service areas, partnering opportunities (e.g., with CCG Marine Radio) and fiscal realities. The needs of mariners have had a major influence on our dissemination systems--most notably the design of the MSC’s Weatheradio network, which has a significant mariner audience. We recognize that the marine community has unique requirements (e.g., often not having access to cell phones and the Internet when at sea), hence some solutions for public weather information dissemination are inadequate for the marine and ice program.
Strategies for dissemination should be based on broader service strategies, and the MSC is in the process of documenting a new service strategy. Once approved, this strategy will provide guidance with respect to many service directions, including dissemination strategies. In the meantime, MSC dissemination experts continue to work with the telecommunications industry to stay abreast of the latest technological advances, while client relations personnel, including those serving the SMT program clients, interact with client organizations to ascertain how their information–access needs are changing. One example of an outcome resulting through these processes is the redesign and reconfiguration of the Weatheradio network in Atlantic Canada, which is currently under way. These changes will see more region-specific programming from each transmitter location, hence reducing the program cycle (the length of which has been a frequent complaint from mariners).
The MSC is working with our primary collaborator in the SMT program, DFO/CCG, to explore the possibility of adapting its automated voice technology used for Weatheradio to be used for the CCG’s marine broadcast system, in order to improve its efficiency. The MSC is also taking an active role in supporting E-navigation strategies that are being led by DFO/CCG, which will lead to new solutions for marine weather information access and display that are compatible with industry needs and expectations, domestically and internationally. These solutions will go beyond dissemination strategies, and will necessitate some reconsideration of the nature and formats of marine weather information entirely. To that end, the MSC is conducting a small project to convert marine forecasts into an xml format. Such a format opens the door to many innovative ways to disseminate the information, not only by the MSC but also by third parties.
1. The Canadian Ice Service (CIS), a division of the MSC, is responsible for delivery of the activities related to ice information. Excluded from the scope of the evaluation are the services provided by the CIS relating to pollution monitoring, prevention and enforcement. These activities relate to the PAA element 188.8.131.52 – Marine Pollution.
- Date modified: