3
$\begingroup$

I have a time series that I'm fitting models to, using R.

I have chosen an ARIMA model based on minimising the AIC_C values.

The ETS model (ets()) was chosen based on minimising the model accuracy errors and the rolling forecast accuracy errors (in the latter, I chose the best 3 from the former, and the ets() model was competing against hw() and hw() with multiplicative seasonality, assuming that exponential models are comparable).

Now, I'd like to know which of the two best models is overall best?

How do I compare them?

And generally, if I fit another type of model, is there a "rule of thumb" for comparing the performance of different models with each other?

Thanks!

Edit: The ARIMA model's residuals fail the Shapiro test but pass the Box-Ljung test. The ets() model's residuals pass both tests.

Improve this question
$\endgroup$
2
  • $\begingroup$ The best way to compare different models is to use holdout data and compare models using some accuracy measures such as MAPE or MAE using predictions and holdout data $\endgroup$
    – forecaster
    Commented Feb 10, 2015 at 15:28
  • $\begingroup$ Basically the same I did to choose the exponential smoothing model? That is, using increasingly long training sets I compared the errors in accuracy of 6 month forecasts against observed data until I reached the last observation of the time series - starting with a training set with 24 observations, then 25, then 26, ... $\endgroup$
    – SiKiHe
    Commented Feb 10, 2015 at 15:30

1 Answer 1

Reset to default
3
$\begingroup$

If your goal is to forecast the variables, then the pseudo-out-of-sample analysis is a good way to choose the best model. Apparently that is what you've done between the ETS models, but you can add the ARIMA to the horse-race.

You will be interested in analyzing some forecasting performance measures, such as RMSE and MAPE, and choosing the model that minimizes those measures.

Usually in practice you may find similar RMSEs or MAPEs between models. Then, a formal way to test if the models provide statistically different forecast errors is to perform the Dielbold-Mariano test.

Note that the package 'forecast' by professor Hyndman has the accuracy() function for calculating the forecasting measures (both in- and out-of sample). It also has the dm.test(), for Diebold-Mariano test.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.