Assessment of NCEP and Local Area Model Forecasts of Convection and Reflectivity

Kenneth R. Cook, Jacob R. Wallentine, and Kevin R. Rider
National Weather Service, Wichita, Kansas

March 24th, 2010


1.0 Introduction

Convective forecasts are of great importance to those on the Plains due to the severity and frequency of occurrence in this region of the country. Many forecasters use NWP (Numerical Weather Prediction) in conjunction with observational data sets to assess the environment. This is done to ascertain the mode of convection that will occur as well as the time in which it will initiate. As NWP has progressed over the years, fields can now be produced to replicate the reflectivity of radar echoes, giving forecasters a first hand look at modeled storm mode outcomes. Additionally, higher temporal resolution from NWP may give forecasters a leg up in preparing for a high impact events should the beginning and evolution be better depicted.

This study was launched to better ascertain the performance of local and national models with regards to convective initiation (CI) and reflectivity performance. This should give forecasters data that will enable them to better understand the advantages and limitations of both NCEP and Local Area Model (LAM) models.

2.0 Methodology

2.1 Model's Used

Cases were selected from the study domain, which for this study included Kansas and Oklahoma. In Oklahoma, the study area was limited to the region north of I-40. Convection had to initiate in this area. This afforded the inclusion of 2 LAM runs, both using WRF-ARW cores and the Local Analysis and Prediction System (LAPS) initialization. Some additional configuration information on the LAMs:

Information WRFARW16 WRFARW32
Horizontal Resolution 16 km 32 km
Vertical Levels 51 51
Time Steps 1 hour 3 hour
Initialization LAPS LAPS
LBC NAM-WRF GFS
CP Scheme KF KF

A total of 43 cases were examined. NWP models used in this part of the study:

Model Description

NAM-WRF

12km WRF produced by NCEP

RUC40 40km RUC produced by NCEP
WRFARW16 16km LAM (see above)
WRFARW32 32km LAM (see above)
WESTNMM4 4km SPC WRF NMM Core for Western U.S.
WESTARW4 4km SPC WRF ARW Core for Western U.S.
EASTNMM4 4km SPC WRF NMM Core for Eastern U.S.
EASTARW4 4km SPC WRF ARW Core for Eastern U.S.

All of these models have a maximum of a 3 hour time-step. For the second part of this study, which involved only models that produced reflectivity output, the NAM-WRF and the RUC40 were excluded.

2.2 CI Location and Time

CI was identified as first echo on the WSR-88D and was compared to the location of first model QPF accumulation that was deemed associated with CI. A CI location error was determined by finding the distance between these two features. The time error was found by identifying the hour that first echo was observed, rounded to the next hour. Thus making it consistent with the way in which model QPF is accumulated. The data was then placed into a scoring system to better determine the overall performance.

2.3 Reflectivity Evolution and Performance

A subjective assessment was made (1-5, 5 being the best) on NWPs prediction of the system evolution as represented by the reflectivity output. A further determination was made, using the same scale, on the performance of the reflectivity data. To better understand these, the system evolution would characterize the storm mode and any evolutionary processes (e.g. Supercell to QLCS) that took place. The performance was based on the location, speed, and representation of the radar observed reflectivity. It should be noted that the same individual assessed either the evolution or performance to reduce any biases that could have been included in the data. The data was then placed into a scoring system to better determine the overall performance.

3.0 Analysis and Results

3.1 Convective Initiation

3.1.1 Location

CI location errors were computed using 25 mile increment range bins. For example, if an error of 37 miles was observed from the point of CI, it fell into the 50 mile range bin. These results can be seen in table 1 and are plotted in figure 1. From the results, we then scored each based on skill, multiplying a skill category error (1-5, 5 being the least error/most skill) by the number of occurrences in each category. From these resultant calculations, we were able to find where each model fell within the skill categories.

Results are shown in table 2. The top 2 are highlighted. From these data, the NAM-WRF40 and the RUC40 were the best predictors, with an average skill category of just over 4, meaning the average error was between 25 and 50 miles.

Model
25 miles
50 miles
75 miles
100 mi
150 mi
150 mi +
Total
WRFARW32 21 7 6 3 6 0 43
NAM40 22 9 8 0 4 0 43
WESTNMM4 14 9 3 5 5 0 36
WESTARW4 15 9 0 4 5 0 33
EASTNMM4 18 8 8 3 1 2 40
EASTARW4 12 6 5 5 1 2 31
RUC40 25 4 7 2 3 0 41
WRFARW16 17 13 5 2 4 0 41

Table 1. Number of occurrences of model in each distance error bin.

 

CI Location

Fig. 1. Spatial error of the convective initiation location.

Model
5
4
3
2
1
Total
Average
WRFARW32 105 28 18 6 6 169 3.79
NAM40 110 36 24 0 4 174 4.05
WESTNMM4 70 36 9 10 5 130 3.61
WESTARW4 75 36 0 8 5 124 3.76
EASTNMM4 90 32 24 6 1 153 3.83
EASTARW4 60 24 15 10 1 110 3.55
RUC40 125 16 21 4 3 169 4.12
WRFARW16 85 52 15 4 4 160 3.90

Table 2. Number of points of model in each distance error bin. Also shows the total and average number of points for each model.

 

3.1.2 Time

CI time errors were computed using 1 hour increment bins. The results can be seen in table 3 and are plotted in figure 2. From the results, we then scored each based on skill, multiplying a skill category error (1-5, 5 being the least error/most skill) by the number of occurrences in each category. From these resultant calculations, we were able to find where each model fell within the skill categories.

Results are shown in table 4. The top 2 are highlighted. From these data, the local WRFARW16 and the RUC40 were the best predictors, with an average skill category of near 4, meaning the average error was around 2 hours.

Model 1 hour 2 hours 3 hours 4 hours 6 hours 6 hours + Total
WRFARW32 15 10 5 7 3 2 42
NAM40 12 10 10 5 4 1 42
WESTNMM4 5 10 5 7 4 5 36
WESTARW4 2 9 5 5 5 7 33
EASTNMM4 5 10 9 8 7 2 41
EASTARW4 5 7 7 6 4 3 32
RUC40 15 15 7 1 0 1 41
WRFARW16 16 14 7 3 1 1 42

Table 3. Number of occurrences of model in each hour error bin.

 

CI Time

Fig. 2. Temporal error of the convective initiation time.

Model
25 miles
50 miles
75 miles
100 mi
150 mi
Total
Average
WRFARW32 75 40 15 14 3 147 3.50
NAM40 60 40 30 10 4 144 3.43
WESTNMM4 25 40 15 14 4 98 2.72
WESTARW4 10 40 15 10 5 76 2.30
EASTNMM4 25 40 27 16 7 115 2.80
EASTARW4 25 28 21 12 4 90 2.81
RUC40 75 60 27 2 0 164 4.00
WRFARW16 80 56 21 6 1 164 3.90

Table 4. Number of points of model in each error bin. Also shows the total and average number of points for each model.

 

 

 

3.1.3 Final Assessment

To ascertain which model performed best in this portion of the study, the two skill scores were averaged. The results are found in table 5, showing the RUC40 and the WRFARW16 being the best overall performers. A clear demarcation was noted in these performers and are highlighted in table 5. These show that LAMs do provide some local skill and can improve on the forecast if used and configured optimally.

Model Overall Avg.
WRFARW32
3.65
NAM40
3.74
WESTNMM4
3.17
WESTARW4
3.03
EASTNMM4
3.31
EASTARW4
3.18
RUC40
4.06
WRFARW16
3.90

Table 5. Number of points, implying the total combined performance

 

3.2 Reflectivity

3.2.1 Evolution

Reflectivity evolution was subjectively scored according to skill. This "skill" ascertained the ability of NWP to properly indicate storm mode and system evolution to an alternate storm mode if applicable. Table 6 and figure 3 both show the scores, where the higher number indicates more skill or better depiction of storm mode and evolution. From the results, we then scored each based on skill, multiplying a skill category error (1-5, 5 being the most skill) by the number of occurrences in each category. From these resultant calculations, we were able to find where each model fell within the skill categories.

Results are shown in table 7 below. The top 2 are highlighted. From these data, both the SPC NMM cores outperformed the remainder of the models. This may be due to more vertical levels within the model as they are currently configured and run. The ARW cores are currently running with a reduction in the vertical levels in order to get the run completed in a timely fashion to fit into the NCEP modeling schedule.

Model
1
2
3
4
5
Total
EASTARW4
1
7
8
10
3
29
EASTNMM4
1
5
7
14
8
35
WESTARW4
0
6
15
6
2
29
WESTNMM4
0
6
13
11
2
32
WRFARW16
5
14
7
10
2
38
WRFARW32
10
11
13
3
2
39

Table 6. Number of occurrences of model in each evolution bin.

Fig 3. Character of reflectivity evolution .

Model
1
2
3
4
5
Total
Average
EASTARW4 1 14 24 40 15 94 3.24
EASTNMM4 1 10 21 56 40 128 3.66
WESTARW4 0 12 45 24 10 91 3.14
WESTNMM4 0 12 39 44 10 105 3.28
WRFARW16 5 28 21 40 10 104 2.74
WRFARW32 10 22 39 12 10 93 2.38

Table 7. Number of points of model in each evolution bin. Also shows the total and average number of points for each model.

 

 

3.2.2 Performance

Reflectivity performance was subjectively scored according to skill. This "skill" ascertained the ability of NWP to properly indicate a location, speed, and representation of the radar observed reflectivity. Table 8 and figure 4 both show the scores, where the higher number indicates more skill or better performance of the model forecast radar reflectivity. From the results, we then scored each based on skill, multiplying a skill category error (1-5, 5 being the most skill) by the number of occurrences in each category. From these resultant calculations, we were able to find where each model fell within the skill categories.

Results are shown in table 9 below. The top 2 are highlighted. From these data, both the SPC NMM cores again outperformed the remainder of the models, likely for the same reason indicated in the previous section.

Model
1
2
3
4
5
Total
EASTARW4
3
8
10
8
0
29
EASTNMM4
1
6
11
13
4
35
WESTARW4
1
12
8
8
0
29
WESTNMM4
1
9
12
10
0
32
WRFARW16
6
15
13
2
2
38
WRFARW32
14
12
10
2
1
39

Table 8. Number of occurrences of model in each performance bin.

 

Fig 4. Character of reflectivity performance.

Model
1
2
3
4
5
Total Average
EASTARW4 3 16 30 32 0 81 2.79
EASTNMM4 1 12 33 52 20 118 3.37
WESTARW4 1 24 24 32 0 81 2.79
WESTNMM4 1 18 36 40 0 95 2.97
WRFARW16 6 30 39 8 10 93 2.45
WRFARW32 14 24 30 8 5 81 2.08

Table 9. Number of points of model in each evolution bin. Also shows the total and average number of points for each model.

 

3.2.3 Final Assessment

When comparing these data averages, table 10 indicates the best in the reflectivity category, the SPC NMM cores.

Model Overall Avg.
EASTARW4
3.02
EASTNMM4
3.51
WESTARW4
2.97
WESTNMM4
3.13
WRFARW16
2.59
WRFARW32
2.23

 

4. Overall Conclusions

Using NWP in the forecast process to forecast CI and solutions can be difficult. That said, current advancements in technology have afforded us the ability to view a model forecast of radar reflectivity. One can infer from these data storm modes, movement, and evolution. The goal of this project was to give the forecaster a better understanding as to the performance of NWP for convection in the the proximity of the National Weather Service's Wichita County Warning Area.

The results showed that for CI location and time, the NAM40, RUC40, and the 2 LAMs were the best performers with a clear demarcation in performance skill scores. Regarding the performance and evolution of the reflectivity data, the two 4km SPC WRFNMM cores outperformed the remainder of the models.

 


 

 

 

USA.gov is the U.S. government's official web portal to all federal, state and local government web resources and services.