A STUDY ON THE WARNING EFFICIENCY AT THE NATIONAL WEATHER SERVICE FORECAST OFFICE IN INDIANAPOLIS, INDIANA WHILE USING WDSS

 

Kristina Holthouse
National Weather Service Office
Indianapolis, Indiana

 

 I.

 INTRODUCTION

The Warning Decision Support System (WDSS), developed by the National Severe Storms Laboratory (NSSL), is a computer system that uses advanced algorithms on real time National Weather Service Doppler Radar (WSR-88D) data (Eilts et al. 1996). Meteorologists use the output from the WDSS algorithms as an aid during the warning decision making process.

During the spring of 1996, NSSL, in a combined effort with the WSR-88D Operational Support Facility, conducted a test of the WDSS at the Indianapolis National Weather Service Forecast Office (NWSFO IND). This test was held so that the performance and effectiveness of the WDSS could be evaluated during real time severe weather events. E. DeWayne Mitchell has provided a detailed report on this test on the Internet at:

 

http://www.nssl.uoknor.edu/~mitchell/ind_final.html.

 

The WDSS that was installed at NWSFO IND included the Storm Cell Identification and Tracking Algorithm, the Hail Detection Algorithm, the Mesocyclone Detection Algorithm, the Tornado Detection Algorithm, the Damaging Downburst Prediction and Detection Algorithm, and the WSR-88D Precipitation Algorithm.

The purpose of this report is to show that the algorithms in the WDSS noticeably enhanced the warning capability at NWSFO IND. This study examines how warning efficiency improved at NWSFO IND based on how the forecasters interpreted and used the WDSS algorithms operationally, and not based on the performance of the WDSS algorithms themselves.

 II.

 THE TEST

The official WDSS test window was April 15 through May 12. During this period, one warning forecaster used the WDSS workstation while a second warning forecaster operated the WSR-88D Principle User Processor (PUP). The WDSS was used as a supplement to the PUP. Coordination between the two forecasters was necessary during the warning process.

The forecasters were able to operate the WDSS during supercell, mesoscale convective system, and severe squall line scenarios. Large hail, damaging wind and tornados each occurred within the NWSFO IND County Warning Area (CWA) during the test period.

Preliminary severe storm event and warning data (hereafter referred to collectively as Event Data) for April and May 1996 is examined in this work from a verification statistics standpoint, and is compared to Event Data from other time periods to show improvement. The Event Data for all of April is used because the first severe weather outbreak in April in the NWSFO IND CWA occurred after the start of the test period. All of the Event Data for May is examined because the WDSS remained at NWSFO IND past the official test window and was used operationally by the forecasters through the end of May.

In order to measure warning efficiency, several verification parameters are calculated. These parameters take into account different aspects of warning verification including the number of warnings issued, the number of warnings verified, how many severe events had warnings in effect, and how many severe events were missed. The numbers shown in this study were computed from preliminary Event Data and therefore may disagree with the final statistics issued by the National Weather Service Headquarters (NWSH). The equations used to compute the values presented in this work can be found in Section 8.

Two verification comparisons are presented in the following sections to show that the use of the WDSS noticeably enhanced the warning capability at NWSFO IND. The first comparison is of April and May 1996 to the rest of 1996, while the second is of April and May 1996 to April and May of 1995 and 1994. These relationships are made so the numbers may be examined from two different standpoints, one from within a severe weather season, the other from between similar climatological environments.

 III.

APRIL AND MAY VS. THE REST OF THE 1996 SEASON

To show that the warning efficiency at NWSFO IND improved while using the WDSS, the period of April and May was compared to the rest of 1996, January through March and June through December. April and May are labeled as WDSS, with the rest of the year labeled as NoWDSS.

Figure 1 illustrates the False Alarm Ratio (FAR) dropped from 46 percent with NoWDSS to 34 percent with WDSS, and the Percent of County Warnings Verified (PV) increased from 54 percent to 66 percent. This indicates that the forecasters were less likely to over warn while using WDSS. Figure 2 shows a comparison of WDSS and NoWDSS verification statistics, including Probability of Detection (POD), Verification Efficiency (VE), and Critical Success Index (CSI). While using WDSS, the POD increased from 56 percent to 85 percent, the VE from 55 percent to 75 percent, and the CSI from 40 percent to 62 percent. This shows the forecasters were less likely to miss severe events, more warnings were verified, and the overall success rate was higher while using WDSS. For a more detailed look at the verification during the WDSS and NoWDSS periods, refer to Table 1.

Figure 1. FAR and PV for NWSFO IND. See text for details.

Figure 2. POD, VE, and CSI for NWSFO IND. See text for further information.

 

TABLE 1

 

Event Data and Verification Measures for NWSFO IND from the WDSS and NoWDSS periods.

 

 

Total
Severe
Events

 

Warned
Severe
Events

 

Unwarned
Severe
Events

 

Total
County
Warnings

 

Total County
Warnings
Verified

 

Unverified
County
Warnings
WDSS

 

120

 

102

 

18

 

128

 

84

 

44
NoWDSS

 

128

 

72

 

56

 

112

 

60

 

52
 

 

False Alarm
Ratio

 

Percent of County
Warnings
Verified

 

Probability of
Detection

 

Verification
Efficiency

 

Critical
Success
Index

 

 FAR

 

 PV

 

 POD

 

 VE

 

 CSI
WDSS

 

.34

 

.66

 

.85

 

.75

 

.62
NoWDSS

 

.46

 

.54

 

.56

 

.55

 

.40
WDSS is April and May 1996
NoWDSS is January-March and June-December 1996

 IV.

APRIL AND MAY 1996 vs. PREVIOUS YEARS

Another way to exhibit that WDSS improved upon warning success at NWSFO IND is to compare NWSFO IND's performance during the WDSS test period to the performance without WDSS during similar springtime environments. To achieve this, April and May 1996 (WDSS) is compared to April and May 1995 and 1994 (NoWDSS). During 1994, 1995, and 1996, the WSR-88D radar was in use at NWSFO IND.

Figures 3 and 4 show that the WDSS enhanced NWSFO IND's warning capability. Figure 3 shows the FAR decreased while using the WDSS, while the PV increased from 45 percent in 1995 and 60 percent in 1994 to 66 percent in 1996. Figure 4 compares the years by showing the POD, VE, and CSI. The POD with WDSS was 85 percent, up from 61 percent in both 1995 and 1994, while the VE with WDSS was 75 percent, up from 52 percent in 1995 and 61 percent in 1994. The CSI improved from 36 percent in 1995 and 44 percent in 1994 to 62 percent in 1996. Refer to Table 2 for more details.

Figure 3. FAR and PV for NWSFO IND. See text for more information.

Figure 4. POD, VE, and CSI for NWSFO IND. See text for further details.

 

TABLE 2

 

Event Data and Verification Measures for NWSFO IND for April and May 1996, 1995, and 1994.

 

 

Total Severe
Events

 

Warned
Severe
Events

 

Unwarned
Severe
Events

 

Total
County
Warnings

 

Total County
Warnings
Verified

 

Unverified
County
Warnings
1996

 

120

 

102

 

18

 

128

 

84

 

44
1995

 

31

 

19

 

12

 

40

 

18

 

22
1994

 

70

 

43

 

27

 

67

 

40

 

27
 

 

False Alarm
Ratio

 

Percent of County
Warnings
Verified

 

Probability of
Detection

 

Verification
Efficiency

 

Critical
Success
Index

 

FAR

 

PV

 

POD

 

VE

 

CSI
1996

 

.34

 

.66

 

.85

 

.75

 

.62
1995

 

.55

 

.45

 

.61

 

.52

 

.36
1994

 

.40

 

.60

 

.61

 

.61

 

.44
1996 data is for April and May, WDSS
1995 and 1994 data is for April and May, NoWDSS

 V.

CONCLUSION

The verification numbers presented in this paper show that the warning efficiency at NWSFO IND was largely enhanced during the WDSS test period in April and May 1996. Although other factors that affect verification, such as type of storm, level of staffing, and level of training were not considered in this study, the study still takes a thorough look at the success rate from the warning verification standpoint. Since the WDSS was tested in a springtime environment, it is not known how much the use of the system would have enhanced NWSFO IND's warning process during other months of the year.

Since the completion of this test, the Storm Cell Identification and Tracking algorithm and the Hail Algorithm have been included in the WSR-88D Build 9. Based on NWSFO IND's experience with these algorithms during the WDSS test, they should be an enhancement to the WSR-88D system.

VI.

ACKNOWLEDGMENTS

The author wishes to thank to staff of the NEXRAD WSR-88D Operational Support Facility and the staff of the National Severe Storms Laboratory for choosing NWSFO IND as a WDSS test site. The author would also like to thank Mike Sabones, NWSFO IND Science and Operations Officer, Dave Tucek, NWSFO IND Warning Coordination Meteorologist, Ed Berry, Central Region Science Services Division Training and Development Meteorologist, and the staff of NWSFO IND for their assistance and for their review of this paper.

VII.

REFERENCES

DOC, NOAA, National Weather Service, 1987: National Weather Service Operations Manual, Chapter C-72, 16-17, (Issuance 87-01).

Eilts, M.D., K.D. Hondl, M. Jain, J.T. Johnson, E.D. Mitchell, D. Rhue, S. Sanger, G. Stumpf, K.W. Thomas, and A. Witt, 1996: Severe Weather Warning Decision Support System. Preprints, 18th Conference on Severe Local Storms, San Francisco, AMS (Boston), 536-540.

VIII.

APPENDIX

The statistics presented in this report were computed using the following definitions, as described in NWS WSOM 87-1.

False Alarm Ratio =

 

Unverified County Warnings
Total County Warnings
     
Probability of Detection =

 

Warned Severe Events
Total Severe Events
     
Critical Success Index =

 

Warned Severe Events
Total Severe Events + Unverified County Warnings
     
Verification Efficiency =

 

Verified County Warnings + Warned Severe Events
Total County Warnings + Total Severe Events
     
Percentage of Verified County Warnings =

 

Verified County Warnings
Total County Warnings

 


USA.gov is the U.S. government's official web portal to all federal, state and local government web resources and services.