Data Sources and Methods for the Weather Warning Index

4.1 Definitions

There are several definitions of note with regard to the index, indicated below.

An event is an individual instance of a weather or environmental hazard that meets hazard criteria thresholds.

The event time is the time at which the criteria threshold is first met for an event. For alerts where the criteria thresholds are accumulated precipitation, the event time is the time at which the accumulated precipitation total first equals or exceeds the criteria amount.

The issue time is the time at which the alert has been transmitted by the Environment Canada forecaster.

The lead time is the difference between the time that an alert is issued and the event time. For example, if an alert is issued at 09:00 and the event time is 09:30, the lead time is 30 minutes.

The target lead time is Environment Canada’s performance goal for the timeliness of alerts. The target lead time is intended to provide adequate time for the public to take appropriate action when alerted of a predicted event. Other factors such as the predictability of an event and the ability of the public and media to receive notice of the message may influence the actual lead times provided.

Each of the six severe weather warnings that comprise the index has an associated performance target lead time, in terms of issuance of a warning. The target lead times for the six warning types are as indicated below in Table 4.

A hit is defined as "a warning event was forecast and it occurred."

A miss is defined as "a warning event occurred but there was no advance warning provided or no alert was issued."

A false alarm is defined as "a warning event was forecast but conditions did not reach warning criteria."

A correct negative is defined as "no warning was issued and no event was reported."

4.2 Methodology

The index ranges from 0 to 10 and is constructed by taking a weighted mean of timeliness and accuracy statistics. To reduce volatility from year to year, a three-year moving average is used for reporting purposes.

Weighting

The index considers demographic and recreational statistics in its calculations.  More specifically, given the statistic that 20 percent of Canadians engage in marine activities,^[1] a 20 percent weight is assigned to the marine gale component of the index. The remaining severe weather warning types (severe thunderstorm, wind, rainfall, snowfall and freezing rain) are assigned a combined 80 percent weight in the index. In addition, each of these land components is weighted based on its frequency of occurrence during the reporting periods, which also reflects its impact on the overall population.

Timeliness factors

To calculate the average lead time for each warning component, the warning issue time is subtracted from the time when the warning event occurred. The lead times so obtained are then averaged over the year to obtain the "average lead time". Missed events are assigned 0 lead time. This value is then compared to the target lead time for the specific warning type.  The resultant value represents the "timeliness" aspect of the index.

Accuracy factors

The number of successfully detected events (hits), the number of missed events and the number of false alarms represents the "accuracy" aspect of the index. These values are used to calculate the Extremal Dependency Index (EDI).

The Hit Rate is calculated using the number of hits and misses, and the False Alarm Rate is based on the number of false alarms and number of correct negatives (Non-events).  This requires an estimation of the number of periods of time during the year that no warning would be required (non-events). Since the EDI is a ratio of logarithms, it does not matter which base is used.

Hit Rate = Hits / (Hits + Misses)

False Alarm Rate = False Alarms / (False Alarms + Non-events)

Weather Warning Index calculation

The index ranges between zero and ten. It would attain a value of ten if all component warnings meet warning criteria and there are no missed events or false alarms.

The scoring formula for each warning component is as follows:

Case 1

If the Average Lead Time is equal to or greater than the Target Lead Time (ALT ≥ TLT), the index component score becomes

[EDI + (0.5 x (AverageLeadTime/TargetLeadTime – 1) x (1 – EDI))] x 10

Case 2

If the Average Lead Time is less than the Target Lead Time (ALT ‹TLT), the index component score becomes

EDI x (AverageLeadTime/TargetLeadTime) x 10

Assumptions

• All lead times are greater than or equal to zero.
• Individual lead times greater than twice the target lead time are assigned a value of twice the target lead time.

The table shows the Weather Warning Index calculation, timeliness and accuracy statistics and component scores for the three-year moving average 2010, 2011 and 2012 by warning type: rain, snow, freezing rain, wind, severe thunderstorms and marine gale.

Weather Warning Index is the sum of the individual weighted Weather Warning Index component scores: 7.89

^* False alarms are not available for severe thunderstorm alerts. A convective warning bias of 5 is used to estimate the number of false alarms, where:

False alarms = Convective Warning Bias x (hits + misses) - hits

[1] Discover Boating Canada (2007) The economic impact of recreational boating in Canada: 2006 summary report. National Marine Manufacturers Association (NMMA) Canada.