A Seasonal Verification Study of the MRF-Based Objective Guidance for Omaha, Nebraska
Michael J. Fuhs and Cynthia M. Fay
National Weather Service Office
The Global Spectral Medium Range Forecast model (MRF)-based objective statistical guidance was implemented on December 10, 1992. The statistical output was derived using a perfect prog method (Wilks 1995). The guidance consists of daytime maximum and nighttime minimum temperatures, as well as the probabilities of precipitation for 12 and 24 hour periods. This guidance, based on the 0000 UTC model run, is produced once daily and becomes available about 0900 UTC. Forecast parameters out to 192 hours are derived from the model.
Many articles have been written in the past regarding the performance of the MRF, including surface cyclonic and anticyclonic development biases and 500 mb flow. The MRF performance evaluations have been presented by Caplan and White (1989), Livingston and Schaefer (1990) and Bedrick et al. (1994). In addition, details of the MRF initialization, wave parameters and physics, and the NMC's Global Forecast System have been summarized in detail by Kanamitsu (1989) and Kanamitsu et al. (1991). The reader is referred to these discussions for an in-depth look into the skills and biases of the MRF model. As of this writing, the resolution of the MRF is a Triangular Truncation at wave number 126 (T126) through forecast day 7, which is equivalent to a horizontal grid resolution of about 105 km. From day 7 to day 16, the resolution is at T62. The vertical resolutions of this model are 28 levels.
The purpose of this work is to examine the performance of temperature and 24-hour probability of precipitation of the MRF-derived objective guidance for Omaha, Nebraska. The AFOS file heading for this message is FMROAX. One year of data was collected from January 1, 1994 through December 31, 1994, and Omaha's Eppley Airfield (OMA) was used as the official observing station against the FMROAX. Statistical tests were performed to find skills and biases of the FMROAX.
2. DATA AND METHODOLOGY
In a previous study involving the MRF-based objective temperature guidance for Grand Island, Nebraska (FMRGID) by Nietfeld and Skerritt (1994), it was found that the MRF displayed seasonal bias tendencies. For that reason, this study was broken down into a seasonal evaluation to determine possible temperature and precipitation trends for Winter (December-February), Spring (March-May), Summer (June-August) and Autumn (September-November).
Temperature and precipitation forecast of the FMROAX product were tested with OMA's observed values through day 6. This closely corresponds to the Extended Forecast Product (EFP). In addition, the MRF-based objective guidance shows a trend toward normal climatology with forecast periods beyond day 6. This trend is the result of the use of a calibration technique that reduces the mean square error based on previous verification data (Jensenius et al. 1993). This method was tested by subtracting the mean climatological temperature from the mean forecast MRF temperature for day 8. Throughout the year, the number of occurrences where this calculation produced a difference of 5°F or less was 301 (82.5%). Of the remaining 64 events (6°F or more), 41 occurred during winter.
Temperature biases of the FMROAX were examined for days 1 through 6 seasonally. The warm and cold errors were tabulated each day for both the maximums and minimums. Errors were calculated by subtracting OMA's observed temperature from the FMROAX predicted temperature. Section 3.1 will show that there were warm and cold biases for each season.
Statistics were displayed to show how well the FMROAX performed against OMA's observations for each season. The statistical parameters calculated for the high and low temperature errors were the mean, absolute mean, median, range and standard deviation. The mean is simply the arithmetic mean error and is equal to the sum of the N observations (denoted by Xi) divided by the number of observations in the dataset (1).
The mean absolute error (MAE) is the sum of the absolute value of the error of N observations divided by the number of observations in the dataset (2).
In this study, N is the number of maximum or minimum temperature observations (approximately 90 for each season). Xi is the sum of these temperature observations. Median error is the middle number, while the range gives the extreme errors. Standard deviation (SD) shows the variability for each day and is calculated in the dataset (3).
In the computation of these statistics, nighttime corresponds to 7:00 p.m. to 9:00 a.m. central standard time (CST) during the cool season and 7:00 p.m. to 8:00 a.m. CST during the warm season. Daytime corresponds to 9:00 a.m. to 7:00 p.m. CST during the cool season and 8:00 a.m. to 7:00 p.m. CST during the warm season (Jensenius et al. 1993). In the database collected for January 1, 1994 through December 31, 1994, there were 22 cases of cool season highs that did not follow the normal diurnal cycle from OMA's observations. Most of these occurrences were midnight maximums during the winter. For the warm season highs, ten cases were non-diurnal. Similarly, there were nine cases of cool season lows and three cases of warm season lows that were out of the normal diurnal cycle. This is a small percentage of highs and lows that were outside the periods and were kept for continuity.
Beginning April 14, 1994 new maximum and minimum temperature equations were introduced into the MRF-based objective messages. The new equations were based on regression techniques similar to the Model Output Statistics (MOS) approach (like the FWC output derived from the Nested Grid Model), as compared to the old perfect prog equations.
The MOS approach consists of prediction equations constructed on the basis of correlations between numerical weather prediction (NWP) forecasts as predictor variables, and the observed value of tomorrow's predictand. The perfect prog technique makes no attempt to correct for possible NWP model errors or biases and assumes them to be perfect. The observed predictors are used to specify observed predictands, meaning only historical climatological data are used in the development of perfect prog equations (Wilks 1995). The perfect prog equations are developed with predictors observed simultaneously with the predictand.
In the previous FMR perfect prog equations, calibration procedures were applied to remove systematic biases resulting in a calibrated perfect prog. Because of the calibrations, the final output was actually a combination of the perfect prog and the MOS that included seasonal thermal adjustments (Nietfeld and Skerritt 1994). The 1000-500 mb thickness field was the main thermal predictor. Using this thickness resulted in large errors with arctic outbreaks. The new MOS equations contain low-level thicknesses and temperatures as predictors, as well as observed surface temperatures at the earlier projections (Jensenius 1994).
Test results from the National Meteorological Center from the winter of 1992-1993 indicated that the new equations improved the MAE about 1.6°F. After 96 hours, little difference in accuracy between the old and new equations was observed. This fact is partially due to the MOS forecast tendency toward normal climatology with time (Dallavalle et al. 1985). In addition, the forecaster must remember that MOS guidance accuracy is dependent on the performance of the model. If the model handles a situation incorrectly, then the MOS guidance may be erroneous.
The change in equations did not appear to affect the data set continuity for this study. To verify this, the arithmetic mean, MAE and median values were examined for each month during winter and spring. The data sets from January and February (produced from the perfect prog equations) were compared against the data set from December, which used the MOS. Likewise, the first half of the Spring (perfect prog) was compared against the second half of Spring (MOS). The magnitude of the errors did not change between the months using the perfect prog equations versus using the new MOS equations.
Another way of analyzing temperatures from the FMROAX is to examine the forecasted mean daily temperatures for above and below normal trends. The 5-day mean temperature class limits for the 6 to 10 day outlook (Rieck 1979) were used as the guidance for above normal, below normal, etc. Since the FMROAX study was done seasonally, the temperature class limits were averaged for the three months of each season for OMA. The above and above normal, and the below and below normal temperature classes were combined to create above normal (AN) and below normal (BN) categories, respectively. Rieck's definition of the near normal (NN) class was unchanged for each season. The use of these categories gives a more accurate test of how well the FMROAX predicted above and below normal temperature trends.
The mean climatological daily temperature was subtracted from the mean daily FMROAX predicted temperature (FMROAX-Climo). Likewise, the mean climatological temperature was subtracted from the mean daily temperature observed at OMA (OBS-Climo). These two daily values were then compared and placed in their respective categories. (Note: The "Climo" values used were from the averaged 96 to 120 hour period, listed at the end of all MRF-based objective messages.)
The result of this procedure (detailed in Section 3B) compares the frequency that FMROAX-Climo and OBS-Climo were either AN, BN or NN. For example, if FMROAX-Climo was in the AN category ten times for a season, and OBS-Climo was in this category 20 times, then the FMROAX accurately predicted 50 percent of the time that temperatures were in the above normal category.
The final analysis conducted on the FMROAX was an evaluation of the 24-hour probability of precipitation (POP24). In the MRF-based objective messages, the POP24 value is the probability of receiving 0.01 inches or more of liquid-equivalent precipitation during a 24 hour period from 0000 to 0000 UTC (Jensenius et al. 1993). The results are shown in Section 3.3.
3. DISCUSSION OF RESULTS
Figures 1a through 1f show the number of times in which the daily high temperatures predicted by the FMROAX were in error for the temperature thresholds given on each graph. During winter and summer, the forecasted high temperatures displayed a warm bias, for days 1 through 4, approximately two-thirds of the time. Day 5 in the Winter was also warm nearly two-thirds of the time. Table 1A shows the magnitude of the warm biases and how the temperature variability (standard deviation) increases toward day 6. Overall, when one examines the arithmetic mean, absolute mean and median values, the warm bias is about 2° to 4°F for summer and 4° to 6°F for winter.
Conversely, spring and autumn high temperature predictions by the FMROAX were too cold nearly two-thirds of the time for days 1 through 6. Again, temperature variability increases upon approaching day 6. The arithmetic mean, absolute mean and median values show a cold bias of approximately 3° to 6°F for spring, and 2° to 4°F for autumn (reference Table 1A). High temperatures for summer and autumn were better predicted by the FMROAX than winter and spring.
The low temperatures predicted by the FMROAX had fewer seasonal bias tendencies than the high temperatures. Figures 1g through 1k show that the strongest seasonal bias was in the Winter, where the predicted low temperatures were too warm about two-thirds of the time. Another significant bias was in the Summer, when almost two-thirds of the low temperatures for days 2 through 4 were too warm. Day 4 in the Autumn was too cold nearly two-thirds of the time, but overall, spring and autumn displayed no strong warm or cold biases.
Table 1B shows that the low temperatures were also more accurate than the high temperatures with variability's increasing much less dramatically upon approaching day 6. Examining the arithmetic mean, absolute mean and median values, winter had the strongest seasonal bias of about 5° to 6°F too warm. Overall, spring was very accurate, summer slightly warm and autumn slightly cool.
Tables 1A and 1B also show the maximum and minimum temperature errors. The maximum (minimum) error is the greatest temperature error in which the FMROAX was too warm (cold) when compared to the observed temperature. As expected, frequently the largest maximum errors occurred when the FMROAX forecasted temperatures that were too warm during arctic outbreaks or when significant cloud cover or precipitation occurred. Often, the largest minimum errors occurred when the FMROAX under predicted strong warm air advection ahead of advancing frontal systems. Interestingly, the largest minimum errors for the low temperatures were not significantly different from the high temperatures, discounting effects of unexpected solar heating.
Table 2 displays' OMA's observed mean high and low temperatures for each season compared with the mean climatological (climo) temperatures. Except autumn, OMA's high and low temperatures during 1994 deviated only slightly from normal.
Table 3 displays how well the FMROAX predicted NN, BN, and AN temperatures. To explain this information, an example from Table 3 is given. The second column of day 2 for the winter season shows 13 days in the Winter were NN. The FMROAX predicted NN temperatures for 9 of these 13 days. Therefore, the FMROAX accurately predicted 9 out of 13 times (69%), NN temperatures for winter. Explanation of this procedure is important because it shows how well the FMROAX predicted these classes of temperatures. It excludes the variability of analyzing the magnitude of daily temperature biases.
The AN categories in Winter and Summer and the BN categories in Spring and Autumn follow the seasonal bias tendencies explained in Section 3.1. Except days 5 and 6 in Autumn, the FMROAX correctly predicted these classes of temperatures 83 percent of the time. Therefore, if the FMROAX predicts a mean daily temperature in the class that follows the seasonal bias, the chances are good that it will verify. The FMROAX accurately predicted NN temperatures nearly 60 percent of the time for winter and autumn and less than 50 percent most of the time for spring and summer. This is probably due to the extreme temperature variability in spring and the narrow NN threshold for summer. The BN categories in Winter and Summer and the AN categories in Spring and Autumn are opposite the seasonal temperature bias trends. Still, the FMROAX correctly predicted these temperature trends 56 percent of the time.
Most of the mean temperature errors that were at least plus or minus 10°F between the FMROAX and OMA, occurred during winter. The MRF is typically too warm during arctic outbreaks or through times of significant snow cover (Jensenius 1994). Also, melting snow cover often causes significant fog and low stratus that can affect temperatures for several days in eastern Nebraska.
Figures 2a through 5e show a direct comparison of the number of OMA's observed precipitation vs. non-precipitation (no precipitation or a trace) events for each forecasted FMROAX POP24 threshold for days 2 through 6 during each season. The thresholds were defined by grouping the POP24 values into increments of 10 percent, e.g., 20 to 29 percent. This method of analysis was used to determine if any daily or seasonal biases existed.
Ideally, the graphs should show most of the cases of non-precipitation events in the lower percentage thresholds and a few of these events in the higher percentage thresholds. This was generally the case for winter and autumn, but to a lesser extent summer, and not for spring.
During the winter, the FMROAX POP24 accurately predicted precipitation for days 2 through 4 when the values were 50 percent or greater. Autumn precipitation trends were similar to winter's trends. During autumn, days 2 through 4 showed that the POP24 was fairly accurate when values were 30 percent or greater. Spring and summer showed that the FMROAX POP24 was not as reliable in determining precipitation for days 2 through 6. For these seasons, a high number of non-precipitation events (compared to precipitation events) existed even when the thresholds were 50 percent or greater.
The strongest overall trend for the FMROAX POP24 was when the POP24 values were less than 20 percent. In this case, a high number of non-precipitation events occurred. This was true for each forecasted day and every season.
The MRF-based objective guidance for Omaha, Nebraska, FMROAX, was evaluated for a one year period from January 1, 1994 through December 31, 1994. The forecasted high and low temperatures and 24 hour probability of precipitation (POP24) were evaluated for accuracy against the observed values from Omaha Eppley.
The FMROAX forecasted daily temperature results are as follows:
Over 80 percent of the time, the FMROAX correctly predicted the mean daily temperatures for the above normal class in Winter and Summer and below normal class in Spring and Autumn. Fifty-six percent of the time, the FMROAX correctly predicted the mean daily temperatures for the below normal category in Winter and Summer, and the above normal category in Spring and Autumn. The strongest precipitation trend observed was for POP24 values less than 20 percent. For these values, a high number of non-precipitation events existed.
The high and low temperature results of this study are consistent with the findings of Nietfeld and Skerritt (1994) in a nine-month study of the FMR guidance for Grand Island, Nebraska, (FMRGID) from December 9, 1992 through September 9, 1993. It is hopeful that the FMROAX seasonal skills and biases found also hold true for other localities, especially in the Midwest. Without evaluating every station, this is impossible to know, but similar results from evaluation of the FMRGID hint that some continuity exists.
The authors wish to extend appreciation to Cindy Fuhs for her expertise in programming the C software. In addition, we would also like to thank Kathleen Schlachter, WSO Lincoln, Nebraska for retrieving some of the FMROAX data.
Bedrick, M.A., A.J. Cristaldi III, S.J. Colucci, and D.S. Wilks, 1994: An analysis of the accuracy of 120-h predictions by the National Meteorological Center's Medium Range Forecast model. Wea. and Forecasting, 9, 3-20.
Caplan, P.M., and G.H. White, 1989: Performance of the National Meteorological Center's Medium-Range model. Wea. and Forecasting, 4, 391-400.
Dallavalle, J.P., M.C. Erickson, and J.S. Jensenius Jr, 1985: Automated daytime maximum, nighttime minimum, 3-hourly surface temperature, and 3-hourly surface dew point guidance. NWS Technical Procedures Bulletin No. 356, DOC, NOAA, NWS, 14pp.
Jensenius, J.S. Jr., J.P. Dallavalle, and S.A. Gilbert, 1993: The MRF-based statistical guidance message. NWS Technical Procedures Bulletin No. 411, DOC, NOAA, NWS, 13pp.
____________, 1994: New max/min temperature equations to be implemented in the MRF-based statistical forecast messages. NWS National Technical Information Message No. 94-08.
Kanamitsu, M., 1989: Description of the NMC global data assimilation and forecast system. Wea. and Forecasting, 4, 335-342.
____________, J.C. Alpert, K.A. Campana, P.M. Caplan, D.G. Deaven, M. Iredell, B. Katz, H.-L. Pan, J. Sela and G.H. White, 1991: Recent changes implemented into the global forecast system at NMC. Wea. and Forecasting, 6, 425-435.
Livingston, R.L. and J.T. Schaefer, 1990: On Medium-Range model guidance and the 3-5 day extended forecast. Wea. and Forecasting, 5, 361-376.
Nietfield, D. and D.A. Skerritt, 1994: A verification study of the MRF-based statistical temperature guidance for Grand Island, Nebraska. Central Region Applied Research Papers, 13-02, 10-21.
Rieck, R.E., 1979: Class limits for 5-day temperature and precipitation forecasts (tabular). NWS Technical Procedures Bulletin No. 256, DOC, NOAA, NWS, 14pp.
Wilks, D.S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467pp.