The majority of the automatic model selection algorithms like auto.arima and ets (https://robjhyndman.com/publications/automatic-forecasting/) are using information criteria (AIC, AICc or BIC) as opposed to Cross Validation for model selection.

This makes a lot of sense for automatic model selection algorithms as it's way faster and you don't need to design your CV protocol (i.e. CV splits and horizon). These ICs approximate the one-step forecast errors. But what happens if we're interested good forecast performance for longer forecast horizons e.g (monthly data with a 12-24 step horizon)?

Is CV going to significantly outperform Information Criteria for longer forecasting horizons? (to a point where it makes sense to invest the time to use CV instead)

I'm not aware of any papers/articles making this comparison, but I think it makes for interesting discussion!

  • $\begingroup$ I think there may be something about AIC optimizing one-step-ahead but BIC optimizing several steps ahead. This connection has been made somewhere on Cross Validated, but I do not remember where exactly and am not sure how valid it is. $\endgroup$ – Richard Hardy May 14 at 6:59
  • $\begingroup$ Thanks @RichardHardy! the related topics you shared are very useful. No doubt sample size plays a big role here. I'm tempted to test this out on a benchmark dataset (e.g. M3 monthly) just for fun! If I get around doing this I'll post the results here as well. $\endgroup$ – leonidas May 17 at 9:59
  • $\begingroup$ The series of M3 are all pretty short, so you are unlikely to get close to asymptotics there. Consider using M4 data instead. $\endgroup$ – Richard Hardy May 17 at 10:41
  • $\begingroup$ Thanks! also found a reference in Forecasting: Principles and Practice regarding your comment on BIC -> otexts.com/fpp3/selecting-predictors.html $\endgroup$ – leonidas May 18 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.