2
$\begingroup$

I am struggling with a case where I am supposed to calculate optimal forecast weights of two forecasts. We have fitted the models on a training set (time series) and want to calculate optimal weights by minimizing the variance of the errors. It almost seems as though we should calculate the weights using the forecast errors, but then we would in theory use previously unseen observations to calculate these weights.

Do any of you know if we should use forecast error or training error when calculating optimal weights of forecast combination?

$\endgroup$

1 Answer 1

2
$\begingroup$

I would not use the in-sample residuals ("training error") to calculate weights, because that would reward overfitting models.

Instead, I would divide the historical data into two batches, for instance the first 80% and the second 20%. Fit your two models to the first 80%. Forecast both out into the last 20%. Use the out-of-sample errors here to calculate weights. Finally, refit your two models on the entire historical sample, but do not update your weights. Then your weights will be based on a holdout sample, but your models will be trained using all data.

Did I write "finally"? I shouldn't have. Finally, be sure to read Claeskens et al., "The forecast combination puzzle: A simple theoretical explanation" (2016, IJF). The "forecast combination puzzle" they refer to is the possibly surprising effect that unweighted combinations often outperform "optimal" combination weights. One possible explanation is that the process of estimating (!) "optimal" weights will introduce its own variance, which will of course carry through to the variance of your forecasts, and therefore your forecast errors.

Thus, whatever you do, you should compare the approach you choose to simple unweighted combinations.

If you want to get fancy, there are a couple more articles on forecast combination in the IJF.

$\endgroup$
1
  • $\begingroup$ Thank you, this was a very good and thorough answer and I appreciate it very much! Thank you $\endgroup$
    – Analyzer
    Commented Apr 1, 2020 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.