National Weather Service (NWS) forecasters are normally required to use a Probability of Precipitation (PoP) statement in the Zone Forecast Product when measurable precipitation is expected. The use of PoPs has been a subject for debate for many years beginning with Curtiss (1968). Presently, rules governing the operational use of PoP statements are found in Chapter C-11 of the DOC, NOAA, Weather Service Operations Manual (1984). PoPs for specific cities are also used in the Coded Cities Forecast and are then used directly in the national verification program of the NWS (Beasley 1997).
As Murphy (1978) discovered, NWS forecasters have expressed dissatisfaction with this verification procedure. Local inquiries of the forecast staff at the NWS Forecast Office (WFO) in Goodland, Kansas, confirmed Murphy's contention. Namely, forecasters felt that verification based on the evaluation of PoP forecasts at a single point does not accurately measure forecaster skill. It most certainly does not reflect the public's perception of forecast accuracy since many public forecasts are issued for areal locations, not specific points.
Most forecasters at NWSO Goodland favored a scheme that used areal coverage to verify precipitation forecasts. Schaefer and Livingston (1988) summarized statistical work that proved that a PoP is equal to the expected areal coverage of the precipitation. The obvious problem for verifying precipitation in terms of areal coverage is that most locations lack a dense enough network of rain gages to adequately cover the affected areas.
One alternative to using a sparse "ground truth" network of gages is to use WSR-88D precipitation estimates of rain. The use of radar estimates to verify precipitation forecasts is not new. Smith and Smith (1978) compared PoPs and radar estimates of areal coverage across Texas using manually created composites from WSR-57 reflectivity imagery. Naber and Smith (1983) continued that work. France (1990) used similar techniques to explore PoP/areal coverage comparisons across eastern Florida. However, it was difficult to employ the methodologies of these studies for real time verification data since the composite creations were laborious and highly subjective. This paper will document an objective verification scheme developed at WFO Goodland that compares PoPs and areal precipitation coverage as determined by either the WSR-88D or from a network of gages.
The forecast of PoPs across a County Forecast Area (CFA) has been available through the Coded Cities Forecast (CCF) for many years. On the other hand, the acquisition of the verifying areal coverages of actual precipitation was difficult. This changed in the mid 1990s when the Missouri Basin River Forecast Center (MBRFC) began issuing the Mean Areal Precipitation (MAP) product in support of Quantitative Precipitation Forecast verification efforts. NWS Central Region policies on this subject are briefly described in the DOC, NOAA, Weather Service Operations Manual, Part E, Chapter 06 (1977). The MAP product, which is issued daily, can use either WSR-88D precipitation estimates or a smoothed extrapolated plot of gages across a Hydrologic Service Area (HSA).
If the MAP is calculated using a network of gages, rainfall estimates are calculated using the Theissen Method. The determination of Theissen weights can be done by defining a Theissen polygon as being formed by the perpendicular bisectors of the lines connecting stations (Larson 1996). According to the MBRFC (Nunn 1996), the Theissen Method addresses non-uniformity of rain gage distribution by assigning weighting factors to each gage. In addition, small weights are given to gages which are located just outside a basin.
If the MBRFC uses WSR-88D rainfall estimates to calculate precipitation estimates, the MAP product is designated MAP(X). MAP(X) is based on the One Hour Digital Precipitation Array Product (DPA) from the WSR-88D. The DPA has a high data resolution which is transmitted automatically to the MBRFC. In return, the river forecast model uses a 6-hour time step which lumps the hourly values of MAP(X) into a 6-hour gridded data set. It does not use Theissen weights. Total weight is assigned only to radar data and one hour automated rain gage data, unless it is felt that NWS cooperative reporting sites give a more superior readout. If this becomes the case, 24 hour precipitation values would be computed using the MAP function. The MBRFC forecaster uses his or her own discretion, considering elements such as time of the year and reliability of the radar.
An example of the MAP(X) product is shown in Figure 1. It shows precipitation estimates for each of the watersheds in the Goodland HSA. Figure 2 shows the geographic location of each of the watersheds. The MAP and MAP(X) products tabulate 24 hour precipitation amounts in six hour increments. Since PoPs are defined for 12 hour periods, it is easy to compare how much precipitation fell during a 12 hour period by using the MAP(X) products.
NATIONAL WEATHER SERVICE
MISSOURI BASIN OBSERVED MEAN AREAL PCPN (IN)
.B KRF 0308
|401||.00 /||.03 /||.00 /||.00 /||.03|
|402||.00 /||.02 /||.00 /||.00 /||.02|
|403||.00 /||.01 /||.00 /||.00 /||.01|
|404||.01 /||.03 /||.00 /||.00 /||.04|
|405||.00 /||.03 /||.00 /||.00 /||.03|
|406||.00 /||.01 /||.00 /||.00 /||.02|
|407||.01 /||.03 /||.00 /||.00 /||.04|
|408||.00 /||.02 /||.00 /||.00 /||.03|
|409||.00 /||.02 /||.00 /||.00 /||.02|
|410||.02 /||.06 /||.01 /||.00 /||.08|
|411||.02 /||.08 /||.02 /||.01 /||.13|
|412||.04 /||.07 /||.04 /||.01 /||.16|
|418||.06 /||.08 /||.03 /||.02 /||.20|
|422||.01 /||.05 /||.01 /||.00 /||.08|
|429||.14 /||.08 /||.08 /||.06 /||.36|
|430||.14 /||.07 /||.07 /||.06 /||.34|
|431||.11 /||.06 /||.06 /||.05 /||.27|
|432||.05 /||.04 /||.03 /||.02 /||.14|
|433||.02 /||.04 /||.01 /||.01 /||.08|
|434||.01 /||.05 /||.00 /||.00 /||.07|
|435||.01 /||.03 /||.01 /||.00 /||.05|
|436||.04 /||.03 /||.02 /||.02 /||.10|
|437||.02 /||.01 /||.01 /||.01 /||.06|
|440||.05 /||.03 /||.02 /||.02 /||.13|
|441||.04 /||.02 /||.02 /||.02 /||.11|
|501||.00 /||.02 /||.00 /||.01 /||.03|
|502||.00 /||.01 /||.00 /||.01 /||.01|
|503||.00 /||.01 /||.00 /||.00 /||.02|
|504||.00 /||.01 /||.00 /||.01 /||.03|
|505||.00 /||.01 /||.00 /||.01 /||.02|
|506||.00 /||.00 /||.00 /||.00 /||.00|
|507||.00 /||.02 /||.00 /||.01 /||.03|
|508||.02 /||.03 /||.01 /||.01 /||.07|
|509||.04 /||.05 /||.02 /||.02 /||.13|
|512||.07 /||.05 /||.03 /||.03 /||.18|
|514||.04 /||.05 /||.02 /||.02 /||.14|
|515||.07 /||.05 /||.03 /||.03 /||.18|
|516||.04 /||.03 /||.02 /||.02 /||.11|
|520||.02 /||.01 /||.01 /||.01 /||.05|
|526||.01 /||.00 /||.00 /||.00 /||.02|
|527||.04 /||.04 /||.02 /||.02 /||.11|
|528||.04 /||.09 /||.02 /||.03 /||.18|
|529||.01 /||.05 /||.00 /||.01 /||.07|
|538||.03 /||.02 /||.01 /||.01 /||.07|
|539||.01 /||.01 /||.01 /||.01 /||.03|
|540||.01 /||.04 /||.00 /||.01 /||.06|
|541||.03 /||.02 /||.02 /||.01 /||.07|
Once the daily MAP(X) products have been obtained, one can compare forecasted PoPs with actual areal coverage. For example, WFO Goodland compared PoP forecasts for each of the communities in its (CCF) product to the MAP(X) product. This was done by assigning watershed basins that were in proximity to one of the communities listed in the CCF. WFO Goodland issues CCF forecasts for four Kansas communities (Goodland, Colby, Tribune, and Hill City), one Nebraska community (McCook), and one Colorado community (Burlington). Each of these communities are then assigned a number of watershed basins that are located near the site. Figure 2 also shows which basins are assigned to each CCF forecast location.
When the MAP(X) product is received, WFO Goodland computes a percentage of areal precipitation coverage across the assigned basins for each PoP found in the CCF. For example, ten watershed basins were assigned to the PoP forecasted at Hill City, Kansas. If three of those basins received precipitation according to the MAP(X) product, the areal coverage was computed to be 30%. This value was then compared to the CCF PoP for Hill City. Obviously, a PoP of 30% was considered perfect while deviations from 30% led to lower scores.
Contingency tables were then developed to compute a score. The tables used a critical value of 60% for two reasons. First, the National Weather Service's AFOS Era Verification Program (Beasley 1997) uses 60% as a "yes" or "no" cutoff for precipitation forecasts. Second, 60% is the first PoP that a "likely" category in the Zone Forecasts can be used. Therefore, a 60% or greater forecast is termed a "wet" forecast in this study.
Based on the above reasons, the final score for each site was determined by the following equations (where AC = Areal Coverage and PoP = the Probability of Precipitation forecast):
|Areal Coverage >= 60%||Areal Coverage <60|
|PoP >= 60%||100 - (|AC - PoP| * .5)||100 - |AC - PoP||
|PoP < 60%||100 - |AC - PoP|||100 - (|AC - PoP| * .5)|
The numerical final score is reduced by one half for forecasts that either: 1) contain a PoP that is greater than or equal to 60% and more than 60% of actual coverage occurred, or 2) contain a PoP that is less than or equal to 60% and less than 60% of actual coverage occurred. Thus, forecasts which are "wet" score higher points when "wet" conditions occur, and "dry" forecasts score higher points when "dry" conditions occur. Conversely, forecasts are scored with greater penalties when a "wet" forecast is verified with "dry" conditions. Based on the above table, a perfect score is 100. The lowest score possible is zero.
The scores were computed daily by a locally written computer program and were readily available to forecasters. An example of the output is shown in Figure 3. Statistics were also calculated on a quarterly and annual basis.
Forecasters appreciated having precipitation verification scores based on areal coverage rather than point occurrences. In general, they felt the scheme more accurately evaluated the forecasts they issued.
Scores were calculated for 1999. The average score across all sites for the entire year was 87. However, the convective months had a score of 82 compared to 92 in the winter months. This reflects the more general coverage of winter precipitation, but also that forecasters place PoPs in forecasts during convective situations where precipitation does not occur. Approximately 85% of all forecasts in 1999 were above a score of 85, while poor forecasts of less than 45 were made approximately 11% of the time. Again, more poor forecasts occurred during the summer, approximately 7% more than winter forecasts.
|ZCZC TOPWRKAPV 000
TTAA00 KGLD DDHHMM
Figure 3. Example of Daily Output Showing Forecasted PoPS, Actual Coverages, and Computed Scores.
A number of concerns, however, must be noted. The first was the assignment of basins to forecast points. Although the assignments were based on geographic proximity between forecast point and basin location, some assignments resulted in distances in excess of 60 miles from the forecast point. This resulted in occasions when precipitation over basins far from the actual point skewed scores negatively. There were other occasions, however, where scores were impacted in a positive manner for the same reasons. A second area of concern was that even a small amount of precipitation coverage in a specified basin resulted in the same contribution to the score as though the entire basin was covered. Thirdly, the entire MAP(X) process has deficiencies in data quality which could conceivably result in inaccurate scoring. Finally, although forecasters quickly utilized the scores in the early stages of the study, most eventually returned to examining the point specific Brier scores available via the AFOS Era Verification program since that is the nationally recognized standard.
Even given the deficiencies, however, this scheme indicated that there is great utility in using areal coverage for precipitation verification studies and evaluations. The processes for acquiring data necessary for such schemes are getting easier to automate. As actual precipitation data and locally produced forecast data become available through the Advanced Weather Interactive Processing System, verification programs using areal coverage will become even easier to implement.
The authors wish to acknowledge Noreen Schwein and Llyle Barker for their helpful suggestions on the review of this paper.
Beasley, R.A., 1997: AFOS Era Forecast Verification. NOAA Techniques Development Laboratory Computer Program 87-2, 63pp.
Curtiss, J.H., 1968: An Elementary Model for the Interpretation of Precipitation Probability Forecasts. J. Meteor., 7, 3-17.
DOC, NOAA, National Weather Service, 1984: Weather Service Operations Manual Chapter C-11, Zone and Local Forecasts, Silver Springs, 51 pp.
____________, ____________, and ____________, 1977: Weather Service Operations Manual Chapter E-06, Quantitative Precipitation Forecasts Used for River Forecasts, Silver Springs, 4 pp.
France, P., 1990: Utilizing Radar Estimates of Areal Rainfall Coverage As a Verification Forecasting Tool. Natl. Wea. Dig., 15, 17-22.
Larson, L.W., 1976: CALB-MAP: Precipitation Model Calibration Mean Area/Precipitation Computations/Procedure. National Weather Service River Forecast System User's Manual, Chapter II, 6, NWS Office of Hydrology, Washington D.C., 1-13.
Murphy, A.H., 1978: On the Evaluation of Point Precipitation Probability Forecasts in Terms of Areal Coverage. Mon. Wea.. Rev., 106, 1680-1686. <>P>Naber, P.S., and D.L. Smith, 1983: Evaluation of Point Probability Forecasts Using Radar Estimates of Rainfall Areal Coverage. DOC, NOAA, NWS NOAA Technical Memorandum NWS SR-108. National Weather Service Southern Region Headquarters, Scientific Services Division, Forth Worth, TX, 17pp.
Nunn, J., 1996: Personal Communication on computing Mean Areal Precipitation using the Theissen Method. NWS, Missouri Basin River Forecast Center, Pleasant Hill, MO.
Schaefer, J.T., and R. Livingston, 1988: What Does the "Probability of Precipitation" Mean? DOC, NOAA, NWS Central Region Technical Attachment 88-1. National Weather Service Central Region Headquarters, Scientific Services Division, Kansas City, MO, 5pp.
Smith D.L., and M. Smith, 1978: A Comparison of Probability of Precipitation Forecasts and Radar Estimates of Rainfall Areal Coverage. DOC, NOAA, NOAA Technical Memorandum NWS SR-96. National Weather Service Southern Region Headquarters, Scientific Services Division Fort Worth, TX, 17pp.