A look at June 2012 NWS Sullivan Verification

                                                             Issued Friday May 31 2013 12:45 pm CDT

To provide some idea of our degree of accuracy and how much uncertainty there is in the forecasts that we give, we put together some charts that show the forecast verification.  These charts cover last year, the 30 day period in June 2012.

Some background.  It was a very warm and very dry month in southern Wisconsin.  We went into severe to exceptional drought conditions in June and July. 

Madison received just 0.31" of rain (4.54" normal), or driest ever June on record.    The average temperature for the month was 4.6F above normal, or 10th warmest. 

Milwaukee receive 0.90" of rain (3.90" normal), 6th driest June on record.   The average temperature for the month was 5.1F above normal, or 5th warmest.

The first chart shows verification for the High Temperature forecasts.  It starts on the left with Day 1 forecasts (for today), and ends on the right with forecasts made a week out (Day 7).  The black line is the average error in degrees F for the 30 day period.  The green line on the bottom is the lowest error for the month for the entire 20 county area in southern Wisconsin.  The red line at the top is the highest error for the month.  So this should give you an idea of what our typical error is, our best forecasts, and our worst forecasts. 

So on Days 1-3, we are usually within 2 degrees F on High Temperature Forecasts.  That slowly grows to 3 degrees F on Day 4,  4 degrees F on Day 5, and 5 degrees F on Day 6 and 7.  It also indicates that errors can be rather large in the extended period, with maximum errors in the 10-15 degree range on Days 5, 6, and 7.    


 The 2nd chart below is similar but for Low Temperature forecasts.   Typical errors grow from 2 degrees on Day 1 to 3 degrees on Days 2 and 3.   4 degree errors are common on Days 4 and 5, with 5 degree error typical on Days 6 and 7.   Maximum errors of 10 degrees occurred on Days 3-5, with up to 15 degrees on Days 6 and 7. 


 Chart 3 shows information about average error for wind speeds in knots.  For reference, 1 knot=1.15 mph.

Typical errors grow slowly from about 2 knots Days 1-4 to 3 knots in the extended period from Days 5-7.  Large errors around 10 knots have occurred on Days 5-7.  

 


 The remaining charts show verification for our rain forecasts, known as PoP (probability of precipitation) forecasts.  They are based on 12 hour periods (like today or tonight).  You can think of it as the odds that it will rain on your location.  For example, if we issue a 50% chance of showers, what we are telling you is that on 100 days that look like today, it should rain at your location 50 times, or about half the time.  

Something to keep in mind when looking at these charts is the climatology of rainfall in June for southern WI.  It typically rains about 25% of the time.  So if you forecast climatology and have a 20-30% chance of rain every day, you would eventually be rather well calibrated based on history.  So during droughts that frequency may decrease to say 10% and in very wet months it might increase to 40-50%.  You'll notice that many of our forecasts, especially in the extended, tended to verify just 10% of the time and reflects the extreme drought conditions where many areas received less than an inch of rain during the month.   This year, we are going into the month of June 2013 in a much wetter state than last year, which should result in much different outcome.   

These charts show how reliable or well calibrated we are.  Ideally, if we issue 10 forecasts of 50% chance of showers, it should rain half the time or 5 times.  That would make these forecasts statisically reliable.   However, what you'll see is that during a drought, we had a strong wet bias.  This result isn't surprising, as wet bias are common during droughts and dry bias are common during wet/flooding periods.   This wet bias wasn't as noticeable on Day 1, but was very pronounced on Days 2-7. 

Something else you will notice is that as the forecast gets farther and farther away, our willingness to forecast PoPs above 50% goes way down.  Part of that is the typical uncertainty with longer range forecasts.  Another factor is that it is harder to forecast thunderstorms in the summer versus rain and snow events during the colder part of the year.  As you know, thunderstorms can be fickle and there is often lots of variability in rainfall over short distances (it literally might rain across the street and be dry on your side of the street). 

The first chart is for Day 1.  Our forecasts were pretty close to the black line, which is what we are aiming for.  The diagonal black line shows a perfect calibration where the forecast probability of rain would exactly match the percentage times it actually rained.   When the forecast is below the black line, you have a wet bias.  When it is above the line, you have a dry bias.  So for high PoPs like 80 and 90%, it actually rained slightly more often than we forecast. 


For Day 2, the wet bias becomes more pronounced.    All the forecasts with exception of 70% were below the perfect reliability line.  In fact, when we forecast 40 percent, it rained less than 20% of the time.


Here is the Day 3 chart.  It is very similar to the Day 2 chart. 

 


 Day 4 chart shows wet bias.  In fact, when we forecast 60% chance of rain, it rained only about half the time.  The drought likely played a role here. 


Day 5 chart shows the general wet bias, though 60% forecasts were well calibrated. 


 Day 6 chart shows the pronounced wet bias.  In fact, it never rained when we forecast 60% chance of showers.


Day 7 chart shows essentially the same information as Day 6. 


Jeff Craven

Science and Operations Officer

NWS Milwaukee/Sullivan WI 



Return to News Archive

USA.gov is the U.S. government's official web portal to all federal, state and local government web resources and services.