WEATHER FOR ENERGY

How to Intelligently Trade Energy Using Weather Forecasts

Fri, 10 May 2013 20:13:00 -0600

Author: Ria Persad, President of StatWeather (Bio)

When I worked as an Energy Trading Floor meteorologist, there was a period of time every morning when we would evaluate different vendor forecasts that would roll in. This would be in addition to evaluating the outputs of various numerical weather prediction models such as the GFS and the European model (ECMWF).

We were endeavoring to sift through the following 4 items:

What overall behaviors were the models pointing to? This would require examining each model separately.
Based upon what we knew about how the models were performing, what would be our call?
What were the external vendor forecasts saying would happen? This would point to what the markets might do.
How divergent were those vendors? This would point to the certainty of what the markets might do. For example, if vendors are all in agreement, chances are that the markets would trade according to this trend.

This 4-stage process would be repeated operationally each day. I have a great deal of respect for energy and commodities trading floor meteorologists, because this job isn't easy. There are deadlines and tight time constraints, and these meteorologists are usually held accountable for their calls. In my 15 years of both being on the job, consulting, and evaluating lots of in-house forecasts, the best meteorologists are lucky if they can improve upon a 1-15 day forecast by an average of 1 degree over the public and private vendors when it's all said and done. It is notoriously difficult to do, but our job as trading floor meteorologist is to squeeze as much accuracy as we can---because this can amount to millions of dollars for a company.

When I started my company StatWeather in 2009, I had one goal in mind: to help people in energy and commodities make better decisions with less effort. We then set about to develop a streamlined, automated system that anybody could use to accomplish the above 4-stage process. It would help to make the meteorologist's job easier, and for companies that don't have the luxury of a meteorologist, it would serve to inform trading decisions in a reliable manner.

First we developed the software tool, “Forecast Skill.” A trader with this software tool can upload any of their vendor data (including their own in-house forecasts), or use StatWeather's collection of free forecaster data (such as the National Weather Service), and the software will automatically compute and show forecaster errors and biases, averaged over different time frames. From looking at “Forecast Skill”, a trader can see when certain forecasts have been too hot or too cool for extended periods of time. It is not unusual for a forecast to be running as much as an 8 or 10 degree bias for 5 or 6 weeks straight, as odd as this sounds!

Then we developed the software tool, “Forecast Consensus.” A trader can see how their selection of forecasts have been trending as a whole, and exactly where those forecasts are converging or diverging, pointing to possible volatility in the markets. It is a quick reference, but a very powerful tool.

Then we developed the software tool, “Forecast Optimization #1 and #2.” Forecast Optimization #1 will take a collection of numerical weather prediction models and forecast ensembles, and combine them into an “optimal” forecast taking into account StatWeather's internal computations of biases and errors. It does basically what a human meteorologist would do in examining various model outputs, but it makes the process more analytical and robust, keeping systematic track of model biases.

Forecast Optimization #2 will operate on the collection of vendor data which the user has uploaded to the software and present a “vendor optimized forecast.”

Now a trader has the complete picture in a “one stop shop” that he or she can see at-a-glance: the trader knows where their vendors are trending, what the models are doing, how accurate their vendors have been, and what a good or “optimized” call would be. This “optimized” call is generally 25% more accurate than going with a straight consensus of vendor forecasts. Sometimes it's only 20% better. Other times, it's as much as 33% better. But the point is that, this computerized system, without any subjectivity from a human meteorologist, basically gives you the accuracy of what the very best in-house meteorologists in the business could get you, or even better.

What does such a software subscription cost? It is about 25% the cost of hiring an analyst. We hope that you will try out our system and see what StatWeather can do to help you trade better with less work and greater returns!

To learn more about the many things that StatWeather offers, please go to www.statweather.com.

Thank you for reading. Best wishes and happy trading!

Accountability in Weather Forecasting

Mon, 22 Apr 2013 12:30:00 -0600

Author: Ria Persad, President of StatWeather - Bio

In 1997, I had heard that there was a major shipwreck on an oil and gas exploration survey because of unanticipated weather conditions, and 5 people almost lost their lives as a result. I was soon called upon to provide marine forecasts for the oil and gas industry on a similar survey, and that shipwreck always served as a vivid reminder to me that weather forecasting is a serious business: in my case, life and limb were on the line.

My boss and I worked on a variety of methods to try to get at the probabilities of extreme weather occurring. We weren't just interested in a forecast, but in the average error we might be able to expect to what degree of certainty. It took forecasting to a whole new level of statistical analysis so that we could ascertain the risk. Sometimes my company would actually send me on these exploration surveys, into the North Sea and into the Arctic waters, places where my own life would be on the line if my own forecasts were off the mark.

As time progressed, I was forecasting for large trading floors where millions of dollars were on the line, possibly people's jobs and livelihoods—including my own. There was a sense of accountability unlike what I had experienced running supercomputer models at a research lab or figuring out the bounds of climate chaos for a thesis. If my forecasts were wrong, there would be measurable consequences.

I have a great deal of respect for the profession of meteorology and for all those who are in the position of delivering actionable forecasts for mission-critical operations. I would like to raise awareness of the importance of accountability and suggest ways that the profession can better serve their end users.

The dictionary defines accountable as “subject to the obligation to report, explain, or justify something; responsible; answerable.” In areas such as engineering and national defense, including the manufacture of missile systems, the Lean Six Sigma process endeavors to help identify the failure rate of a system. How reliable is the system, and if there are failures above a certain tolerance, what are the root causes, and what can be done to fix them? When we founded StatWeather, we decided to transfer this process to the realm of weather forecasting. Just as the failure of a missile system could cost thousands of lives, poor forecasts could cost just as many lives and fiscal consequences. We felt that it would be a disservice at best, and dishonest at worst, to administer forecasts if people did not know exactly how accurate they were.

Back in 2011 and early 2012, we had a few seasons in a row where our long-range forecasts were consistently inaccurate for the state of Florida and for the Pacific Northwest. Having poor forecasts for Florida was particularly embarrassing, because StatWeather is headquartered in Florida, and we weren't getting our own state right. We felt it would be unconscionable to market our long-range forecasts to any customers in Florida, Washington state, or Oregon, until our track record improved in these areas. This meant that all our friends at Florida Power and Light, Tampa Electric, People's Gas, and Iberdrola Renewables just had to wait. We figured out a better model using predictors with stronger signals in these geographical areas, and we got to where Florida and the Pacific Northwest became a strength. Now we gladly serve customers in these localities. We regularly reported back to our customers and potential customers regarding this issue and what steps we were taking to improve.

Meticulously tracking our accuracy can be tough on the nerves, but is one of the greatest things that helps us to improve and get better, because it shows us where we're off, by how much, how often, and it pushes us to perform and relentlessly serve our customers. It's no good to simply cherry-pick a few locations and only show where we're accurate, or use the excuse, “That was a tail risk” or “Nobody else got it right.”

Back-testing is a useful tool for forecasters to show their customers how well their models have performed. Back-testing is a historical simulation that would show you what the model would have given you at a past date. Then the accuracy of this result can be calculated in hindsight. Ten years of back-testing can give a solid representation of a model's historical accuracy. We like to provide our customers with as much back-testing as they care to digest. Retrospective analyses and regular skill-reporting provide the basis for ascertaining forecaster biases, strengths, and weaknesses according to location and time frames.

Transparency and full disclosure makes us more vulnerable to criticism, but it can also make us more competitive. The weather forecaster is often the butt of jokes, as people like to quip about how hit and miss weather forecasts are. They mock us tongue-in-cheek and say, “Only a weather forecaster can be wrong all the time and still keep his job.” I believe that if more and more forecasters publish their accuracy to their end users, it would expose flawed methods, force the bad forecasters to get out of the business, bring merit to the good forecasters, and raise the reputation of the profession. It hurts the industry, it hurts the customer, and it hurts the public when forecasters are cagey about their track record.

Here are some things to consider when evaluating the risk of a weather forecast:

What are the mean absolute errors and biases of the forecaster's track record for given time frames and given locations of interest?
What is the methodology underlying this forecast?
How does this methodology perform in a back-test for the last ___ years?
How has this methodology performed in recent tests or how has it performed operationally in real-time?
How often does this forecaster present forecast verification to their end users?
Is this verification sufficient to make actionable decisions? For example, does the forecaster provide 68% (one sigma) and 95% (two sigma) confidence intervals or measurable risk metrics so that the end user can hedge his or her risk?

The forecaster is not in a position to be right 100% of the time, even with the best of models. From a purely objective standpoint, there is no shame in being wrong some of the time, because we will never fully account for all of the forces of nature. Accountability does not mean that there needs to be an expectation of 100% perfection; accountability means looking a customer squarely in the eye and honestly reporting where and when you blew it without deflecting. Any forecaster with this kind of chutzpah deserves my respect.

To learn more about StatWeather and to access our accuracy reports, please visit www.statweather.com

Are Long-Range Weather Forecasts Any Good?

Thu, 14 Feb 2013 14:52:00 -0600

Article by Ria Persad, President of StatWeather

In 1997, it was challenging to forecast bad weather in the North Sea for oil and gas operations 5 days in advance. We looked at satellite altimetry data, buoy data, onshore land forecasts, NOGAPS models, climatology---anything that could help us get a handle on when to plan our operations.

In 1999, the Power Traders at Enron wanted me to tell them when there would be a heatwave in Cincinnati...30 to 45 days in advance. At the time, I was too young to think that this was in any way “impossible.” After all, CRAY supercomputers are modeling climate change 30 years into the future.......so what's 30 days? No biggie.

So how is it that the weather people can't figure out what the weather will do 10 days out, yet we're modeling climate 30 years out? And how can anybody claim that they can forecast the weather months in advance?

The answer to this question lies in one word: PRECISION.

Out to about 2 days, we can forecast the weather with hourly precision. Beyond this point, we can forecast with daily precision out to 10 days. Forecasts that are in the 2-week to 4-week range (e.g., ECMWF weather model) can tell you general trends within a few days of occurrence within certain probabilities. Beyond the 4-week range, now things get interesting.

In the longer range, we move into what are called “probabilistic forecasts.” This means that you will have a statistical range, and a probability that the temperature (or precipitation) will fall within that range. This is how extreme events can be estimated in the long-range.

For many years, people have used simple techniques of analog forecasting, whereby you look back in history to find a similar weather year and use what happened in the past to predict the future. The limitation to this method is that depending upon how well you can match up the pattern, this will govern (in theory) the chances of that particular outcome.

There are more sophisticated methods of pattern-matching which might exploit when a phenomenon or far-away weather pattern will trigger a weather event at another time or location. This can sometimes help to target when an extreme event is coming your way. One form of this kind of forecasting is what meteorologists call “teleconnections.”

There are also tell-tale signs of impending events in the long-range. For example, when there is a drought, and if temperatures are hot, then the drought will likely continue in a feedback. Hotter temperatures lead to more drought, more drought leads to less evaporation and moisture and cloud cover, which leads to more heat. So long-range forecasters, whether at NOAA's Climate Prediction Center or at my own company StatWeather, can calculate a long-range drought prediction.

When Accuweather came out with their 25-day forecasts, they were under fire by many in the community who felt that such a thing is an impossible sham. Depending upon their methodology, I would say that long-range weather forecasts can be helpful if people realize that the forecast may point to a weather direction or a most probable weather outcome based upon a model, historical evidence, or statistical analysis.

The means of forecasting a heat wave 30 or 45 days in advance require efforts in the realm of climatology rather than short-range dynamical modeling. Dynamical modeling of the atmosphere (which is used for short- and medium-range forecasts) is like using a microscope. Climatology is like using a telescope. One looks at fine-tuned changes to predict weather at a smaller resolution. The other looks at large-scale climate changes to predict long-range patterns.

They are both equally as scientific and valid in my book, but different toolsets to help us understand weather at different timeframes and granularity.

Are long-range forecasts any good? They can be, as long society can recognize their capabilities and limitations.

If you would like to receive StatWeather's long-range forecasting skill reports, please e-mail service@statweather.com.

Beating Climatology: The Boundaries of Seasonal Forecasting

Sun, 20 Jan 2013 21:06:00 -0600

Article by: Ria Persad, President, StatWeather

Occasionally I hear a long-range or seasonal forecaster say, “We are better than climate normals [climatology] 80% of the time.” In other words, they are saying that their forecast is more accurate than going with, say, a 30-year average or a 10-year baseline average some 80% of the time.

The fact of the matter is, this is a mathematical absurdity. Why? If a long-range or seasonal forecast can take on 3 equally probable possibilities (Below Normal, Normal, or Above Normal), then, in general, there is a 1/3 probability of the weather being “normal.” In other words, climatology will, by definition, predict the climate pattern about 33.3% of the time. If a forecaster were to call every weather pattern accurately, he or she would not be able to “beat climatology” or “beat climate normals” when the weather is actually normal, which is about 1/3 of the time. Therefore, the forecaster can only possibly “beat climate normals” the remaining 66.7% of the time when the weather is NOT normal.

If a forecaster calls the weather correctly 80% of the time, then this in general means that they are beating climatology 80% out of the 66.7% of the time that it's available or possible to beat climatology. 80% of 66.7% is 53%.

Thus, a forecaster who is accurate 80% of the time will beat climatology on average of 53% of the time.

What if you have a forecaster who decides to simply pick at random whether the climate will be “above normal”, “normal”, or “below normal”? This “random forecaster” will have a 33.3% probability of having a correct call at any given time, which means that he or she will beat climatology 33.3% out of the 66.7% of the time that it's available or possible to beat climatology. This means that the “random forecaster” will beat climatology about 22% of the time.

Thus, a random forecast will beat climatology on average of 22% of the time.

A common misconception is that if a forecaster beats climatology less than 50% of the time, then he or she is worse than flipping a coin. This would only be true if forecasts were binary (either “above normal” or “below normal”). The fact is that a forecast can also run “normal”, giving 3 possible outcomes. Therefore, if a forecaster beats climatology more than 22% of the time and also is as accurate as climatology more than 11% of the time, then that forecaster has skill! (In other words, the forecaster is worse than climatology less than 2/3 of the time, which is beating the odds.) This is a very profound result from the statistics of having 3 choices.

So, to recap, a “perfect forecaster” will beat climatology 66.7% of the time. A forecaster with 75% accuracy will beat climatology 50% of the time, and a random forecaster will beat climatology on average of 22% of the time.

Thus, if your in-house meteorologist is doing better than climate normals half of the time, that's pretty darn good.

If you would like to view StatWeather's forecasting accuracy for the Winter 2012-2013 to date, please e-mail service@statweather.com. To learn more about StatWeather, please go to www.StatWeather.com.

When Are Consensus Forecasts Better?

Wed, 19 Dec 2012 06:00:00 -0600

Article By: Ria Persad, President, StatWeather <click for bio>

Whether you are forecasting the weather, stocks, elections, or sports, the question of when to go with a consensus is critical to maximizing gains. This article will present some rules of thumb and the quantitative reasoning behind them.

Let's say that weather forecaster A is usually too warm by 2 degrees. Forecaster B is usually too cool by 2 degrees. It would make sense that the average of forecasts A and B would be more accurate than either of them alone, because their errors cancel.

What if forecaster A consistently hit the nail on the head, but forecaster B consistently was wrong? Averaging these two forecasts would water down the accuracy gained by only going with forecast A. In this case, just going with the better forecast would be the better plan.

So on a very basic level, if we have forecasts that demonstrate “comparable” accuracy, then combining them can result in some cancellation of errors and, hence, a better forecast. However, if one forecast is consistently the “winner”, then combining it with a far inferior forecast is not a good strategy.

However, determining whether the heuristic skill of two forecasters are “comparable” is not always a straightforward task.

What if the average error of forecaster A is +/-5 degrees, and the average error of forecaster B is +/-6 degrees, but forecaster A consistently runs a -1 degree bias (forecaster A is consistently off-center by 1 degree)? Who would you say is the better forecaster? Should a consensus be used? In this event, forecaster A's predictions can be offset by +1 degree. This is what some climate modelers do if they see a warming or cooling trend---they will offset their forecast for bias to center their forecasts. But even with this calibration, should forecast A and forecast B be combined for a better forecast?

In Statistics, there are tests which determine whether one distribution is significantly different from another, or if one group of forecasts is statistically “more” or “less” accurate than another group. Perhaps the difference between the two sets of forecasts is small enough that it could simply be due to random variation. It is not a trivial question, and at best we can come up with a probability that determines whether or not the two forecasters show “comparable” accuracy or if their differences are statistically significant according to a certain threshold of significance.

Empirical studies show that where performance of forecasts is “comparable” or within a certain degree from each other, then a greater number of combined forecasts renders the greatest accuracy. If fewer forecasts are used in the consensus model, then each forecast is “mission critical” and has to “do its job”, so to speak. One bad forecast can ruin the bunch. But if there are a great many forecasts in the mix, one single bad forecaster is not going to impact the whole quite as much.

It then boils down to an optimization problem involving (1) the number of forecasters and (2) the performance of each forecaster in terms of error, which requires a historical analysis of performance of each forecaster. Theoretically, an ideal situation is going with a single forecaster that is right 99.999% of the time. A second-best scenario would be to go with a consensus of one million forecasters using one million independent methods who are each right 80% of the time (note the key being “independent methods”, so that they all don't have the SAME bias nor are they all erroneous at the SAME time), so that the average of their combined forecasts approaches 100% accuracy.

Most of us do not have the luxury of either of these extremes, so the work is in finding the “happy balance” somewhere in between. In addition to our signature long-range forecasts, at StatWeather we measure the accuracy of numerous forecasters over time—over decades, in some cases—and have developed a system of metrics which categorizes the skill of forecasters and models. (We only track what is publicly available or where others give us permission to do so.) This then means that at any given time, we can generate the optimal combination of forecasts and/or models to produce a consensus forecast that is the most optimized for accuracy at any given location. Many industries such as energy trading, utilities, commodities, and risk management benefit from this optimization.

StatWeather's computer program is able to say, based upon past performance metrics, what the most accurate consensus or combination of forecasts will likely be. It might be a subset of forecasts or models, or it might be a single model, or perhaps all models in combination at any given time for any given location.

In my last blog article on Energy Central entitled, “Exactly How Accurate Are Weather Forecasts?”, we established that just going with two or three private vendors doesn't necessarily optimize accuracy. Sometimes publicly available forecasters are more accurate than private vendors, and vice-versa. In any kind of hedging strategy, there is always risk. The key is to have the resources to be able to gain insight into that uncertainty and quantify the risk. Replacing subjectivity with analytics—or attempting to simulate a human-based process of evaluating different forecasts by a more robust, predictable system— is at the heart of any algorithmic system that identifies arbitrage and maximizes returns.

For more information about Consensus Forecasting or Long-Range Forecasts, please contact service@statweather.com.

Exactly How Accurate Are Weather Forecasts?

Wed, 24 Oct 2012 09:31:00 -0600

My name is Ria Persad, and I am the President of StatWeather. For the last 13 years, I have been tracking forecaster accuracy--both publicly available weather forecasters and private vendors. Accurate weather forecasts are vital to the energy and commodities sectors where a one degree shift can mean differentials of millions of dollars.

When I first started tracking forecaster accuracy, The Weather Channel and Accuweather were tied for the bottom of the heap. Over the years, they have both improved their forecasting, and now are frequently more accurate than even the private vendors.

Anymore, it is really a toss-up whether the public or private forecasters are more or less accurate. Certain weeks or months, certain forecasters will lead the pack. Other months, they will all demonstrate considerable bias and help to throw off an entire market. StatWeather tracks these shifts so that companies can recognize arbitrage opportunities in the marketplace.

Short-range forecasts in the 1- to 5-day range are, on average for the United States, about 2.5 degrees off (give or take), which means a 5-degree error range. This is an average; the daily variability can be as much as 10 or 15 degrees off. In my 13 years of tracking, I have seen major forecasters deviate by as much as 20+ degrees for a day-ahead forecast. I have also seen forecasters consistently be too warm by 10 degrees for a particular city for an entire month straight.

This brings me to another point. Do public or private forecasters check themselves? The answer is, not enough of them do, or if they do, it's not consistent; and then fewer yet will release this information to their clients or to the public.

If you were to hire a baseball player for a world-class team, wouldn't you want to know their batting average? Smart companies will keep daily record of how the forecasts they use are performing--including their in-house support. When millions of dollars hinge upon a few degrees difference in temperatures, this information is vital.

In the early 2000's, I used to track the accuracy of the in-house meteorology support at Duke Energy and compare it against the private vendors. The in-house support would be sweating bullets every time I would run these reports, but in the end, the in-house support was more accurate than the vendors. It gave the traders reassurance that they could reliably trust their in-house support.

No private forecaster should be trusted without the hard-and-fast mathematical facts of their performance. Anecdotal evidence is not evidence, and nothing can really replace analytics. No matter how friendly, how cocky, or how popular a forecaster might be, their accuracy might not be any better than a free forecast.

So in all fairness, what is StatWeather's accuracy? StatWeather specializes in long-range forecasting, and over a 10-year period of record, our year-ahead forecasts have been as accurate as a 1- to 5-day forecast, which has been a shocking conclusion. In other words, if we forecasted 55 degrees Fahrenheit for Cincinnati for next March (6 months out, say), the actual temperature would likely range from 53 to 57 degress. This beats using climatological normals. If a customer wants to see weekly accuracy for any city in the country, the report is there with 100% transparency.

When I started the StatWeather company, I wanted to bring our reporting standards up the level of some common engineering standards such as the Six Sigma Black Belt, applying process improvement and quality control standards that you might see for the manufacture of space shuttles or weapons systems. Transparency leaves us more vulnerable to criticism, but ultimately, if our system works, companies soon recognize the benefits. If our system isn't working well, then it's back to the drawing board for StatWeather, and we improve our process.

To view StatWeather's accuracy reports for last Summer 2012, last Winter 2011-2012, or for the last 10 years, email service@statweather.com.