2
$\begingroup$

I have a forecast model I am developing that uses some very unreliable input data, missing data (due to sensors or comms failures) is the rule, not an exception. The quantity being forecast is a daily average over a region, so the average of all hourly measurements from all sensors. This forecast is made just before each day boundary by a human forecaster, assisted by a model (I am working on an improvement to our existing model).

Now, the main part of the model takes recent history of daily averages and uses some fairly standard time-series forecasting techniques to predict the next day. This works okay, however it is well known the that last few hours of each day are particularly strong predictors of the next days value and the human forecasters have been using this knowledge for some time. In order to incorporate that technique into the model, I take the daily value time-series prediction and combine it with the last few hours of data for the day just ending, then train weights for each of these for a weighted average ensemble prediction (I optimise over RMSE). With the weight for the daily model fixed at unity, the sum of the hourly model weights typically come to around 0.5-0.8 or so. I retrain the weights each day using only the most recent 100 days of data because there are biases in each sensor (due to the underlying physics of the system as well as sensor issues) that change over time.

The reason I used a weighted average and not some other much more powerful technique is due to the missing data. Most days have about 5-10% of sensor data missing when the forecast is being made (but not the same 5-10% each day). I figured a weighted average should be robust in this case, with the weights (hopefully) corresponding to the relative predictive power of each model.

Now, this worked pretty well however I thought I could do better. Knowing that the hourly data has unique (but unknown a priori) biases, I thought I could fit a simple linear model for each hour for each station, using the last 100 days of data for that station and hour as the independent variable and the daily average for the following day as the dependant variable. Then I'd take the predictions from each of these very simple models, include the daily model based predictions and take a weighted average of those. In other words instead of just using the raw hourly data, I'm using a model trained using that data. This would surely produce a better result. But no, the weighted average forecast performs more poorly in this case. I simply cannot understand why.

I checked and for an un-weighted average, using the predictions from the linear models produces a better result (as you'd expect) but once I train the weights, the model using just the raw data performs better. Interestingly I find that in the linear model predictors case, more of the weights (around 60-70% of them typically) actually get fixed at zero in the optimisation, whereas for the raw data approach less than 50% get fixed to zero (I am using L-BFGS-B minimisation [in scipy in anyone is interested]). It was actually that fact that so many got fixed to zero in the raw data case that lead me to try the linear fit approach. I figured that the badly biased sensors were being ignored and if I fixed the bias then perhaps that would be able to contribute more meaningfully.

I'm not sure if I'm just doing something a bit silly, or if there is a fundamental concept that might explain this. When might you expect an ensemble forecast to perform better with worse input models?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.