On forecaster confidence and accuracy...
As the head of a weather company, StatWeather, I often get the question, "How confident are you in that forecast?" Techniques for determining the certainty of a forecast coming to pass can range from subjective, touchy-feely guesses to highly analytical, systematized methods. Here is a simple checklist of 6 items that I use to evalute the degree to which I would trust a particular forecast and to attach a "risk" for a specific forecast coming to pass. (You don't need to be a meteorologist to apply this list.)
1. How has this forecaster been verifying lately? No matter how "confident" a forecaster might be in their forecast, if they're almost always wrong, then that forecast is just plain risky!
2. Does this forecast keep flip-flopping or are the model runs consistent? If the forecast keeps changing every 6 hours, then I would wait to see some consistency before trusting that weather forecast.
3. Do the ensemble outputs show good agreement? In an ensemble-based forecast, if the standard deviation or the spread of the ensembles is low, then there is a high chance that, even if the current weather conditions are a little different than expected, it's not going to throw off the whole forecast going forward.
4. Is this forecast consistent with the forecast for nearby/surrounding locations? If I gave you a forecast that it will be 100 degrees in New York, but every other city in the Northeast is forecasted to be 65 degrees, then I would be very careful about that New York forecast! There can indeed be micro-climate effects, but it's good to compare an individual city against the overall region just to be sure.
5. Does this forecast agree with government models? Assuming that government models have been reasonably trustworthy, then you can check to see if the forecast in question is lining up with other models.
6. Does this forecast agree with top-performing forecasts (or DISAGREE with poor forecasts)? Let's say that my competitors have been on a losing streak all winter. If my forecast DISAGREES with them, then that should be a good sign that my forecast is more likely to be right!
If you can answer these 6 questions whenever a big decision rests on a weather forecast, you will have a very good gauge of the forecast risk and, in turn, know if it is wise to trust a particular weather forecast.
If you would like to learn more about forecast risk, probabilistic forecasting, and long-range forecasts, please e-mail firstname.lastname@example.org or go to www.statweather.com.