With respect to doing a trial of TESLA’s electricity and gas demand forecasting models, there are some observations and pitfalls which we would like to discuss with you:
Length of Trial
Electricity model assessment is best done over a period of at least sixty days. Ideally, the trial period should span over a time period in which a wide variety of weather is observed. For instance, when we won the competitive benchmark to be designated as the preferred demand forecast provider for Transpower in New Zealand, the trial lasted for six months, (preceded by a development period of two months). The trial for Saudi Electricity Company (which we also won) lasted five months. For custom models done for a client, or for our public data models, we always offer a free trial period of at least sixty days. Any model can perform better than any other model for a few days, just by chance. A shorter trial period does not give you enough data on forecast quality to assess relative accuracy with confidence.
Number of Models
TESLA strongly prefers trials that involve the entire set of models that the client will eventually need. This allows you to see how the entire system of models will behave when put into service. It also protects you from a vendor putting a large amount of effort into a single model, winning the contract, then delivering a set of models done by junior staff that are not at the same level of quality. Our New Zealand trial involved 180 models. Our Saudi Arabian trial involved seven models. In TESLA’s trial for npower in the UK, we developed 42 models for two different trials (28 power and 14 gas). We strongly suggest that if you need twenty models, then the trial require the vendors to supply all twenty models.
Real-time v. Batch-data Updates
TESLA is designed to be most effective in a real-time or near-real-time environment. We strongly recommend that the trial be conducted with load figures that are available in real time, and with actual weather data (and forecasts, if possible) being available in real time as well. TESLA has done trials in the past with load data only being updated once a day (typically just after midnight). We have won several such trials. However, one of our major strengths is intraday forecasting using real time data, and we greatly prefer being able to demonstrate that.
Live Trial v. Holdout Samples
TESLA has participated in trials where a portion of the load data history is withheld. This is called a “holdout” sample. Models are then run over the (generally very short) holdout sample period, and the performance of the models is then compared. The ability to assess real time model performance is lost in this sort of approach.
We urge you not to do this sort of trial. Any load data that has already occurred can be obtained by a diligent researcher, legitimately or illegitimately. If you want to have an impartial trial, it is important to base your assessments only on the model’s ability to forecast demand figures that are not known at the outset of the trial. It is also important to make load and weather data available at the same time or before you publish it over the web or send it to any vendors, contractors or other third parties.