I need to compare the forecasting ability of different specifications of the ARMA-GARCH model. I would like to compare the model by valuating for each model in-sample forecast and out-of-sample forecast performance.

I know in-sample-forecasts are those generated for the same set of data that was used to estimate model's parameters, whereas out-of-sample forecasts are those generated for the set of data not used to estimate model's parameters.

Since for out-of-sample forecast the multi-step-ahead foecasts could not be ideal as the forecasting horizon increases, one away around this problem is to use a recursive or rolling window. In the latter cases, recursive or rolling, I would re-estimate the model's parameters several times, so I don't have just one estimated model, and I guess estimated parameters will be varying a bit between each estimation.

Hence my question is: how do I evaluate the in-sample forecast given that I estimate the model several times? Could I estimate, for in-sample fit evaluation, the models using all the sample data?

My idea was:

  1. Estimate the models using all the data.
  2. Compare the in-sample performace using information criteria as AIC, BIC and AICc.
  3. Rank the models.

For out-of-sample forecast I would

  1. Divide the data set into two subsamples and then run a rolling window forecast.
  2. Then evaluate the forecast performance using MSE, MAE, DAC and Value-at-risk violations.
  3. Rank the models.

Finally, I would compare the ranking from in-sample forecast with the ranking from out-of-sample forecast and make my conclusions.

Does it make sense?

  • 2
    $\begingroup$ Your plan sounds fine. You only need to measure in sample performance once with the full sample. Then re-estimate with rolling/recursive windows and collect out-of-sample forecast statistics...that is fairly standard. $\endgroup$ Mar 11, 2016 at 21:59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.