I am using forecast() package in R to predict future values. I have a time series data for approx 6-7 years.

First, I split the data into training set and test set. Test set contained values of the last 12 months of original data. Then, applied ets() and auto.arima() on the training set. This can be done by iterating 12 times using 1-step ahead OR by running once to get 12 points at the same time, which is my question no. 1

Then, the output received is compared with the test set to check which of the two (ets or auto.arima) gives minimal error in comparison with test set. The selected model class is then applied on the original data set to predict future 12 values. So, here arises my question no. 2

In Short, I have two things to do

  1. Compare accuracy measures for test set of ets and auto.arima to select one
  2. Predict future values using the derived model from the first step


  1. Should I use one-step ahead or 12-step ahead forecasts to generate data for test set?

  2. Should I use one-step ahead or 12-step ahead forecasts to predict future values?

  3. Should the method be same in both the steps mentioned above?

Kindly help.

  • 1
    $\begingroup$ What advice you are seeking here is unclear. It is not clear whether you have already selected the better method or need to do that: different parts of your post appear contradictory. 1-step and 12-step (and others too) could both be interesting and useful. Comparing different methods is usually a good idea, whether they produce similar or different results. It doesn't make much sense to compare 1-step from one method with 12-step from another; I imagine most readers would expect both. In short, I don't think you give enough information to enable us to advise much on your decisions. $\endgroup$
    – Nick Cox
    May 23, 2013 at 10:42
  • $\begingroup$ made changes to the question and explained in more detail now..hope this helps $\endgroup$
    – priyaj
    May 23, 2013 at 10:56
  • 1
    $\begingroup$ This is now for time series experts. I am not one, but I see no reason to revise my comments. $\endgroup$
    – Nick Cox
    May 23, 2013 at 11:02

1 Answer 1


Your questions are still not very clear, but if you want to compare the forecast performances of the two methods maybe you can do it like this?

tr.data<-window(data,statrt=c(1973,1),end=c(1977,12)) # Training data
test.data<-window(data,statrt=c(1978,1),end=c(1978,12)) # Test data



Forcasts with the models:


Extracting the 12-step-ahead forecast errors from the models:


Calculation of MSE for the forecasts:

72275.59 => MSE of ets-forecast

83421.72 => MSE of arima-forecast

Application of dm.test from forecast-package to compare predictive accuracy:

=>p-value = 0.6145 
although  the MSE of ets is smaller, hypothes of equal performance cannot be rejected for the 12-step forecasts

You can also compare the errors within the training period:

p-value = 3.548e-05 
=> the hypothesis of equal performance of one-step forecasts is rejected!

If you modify the alternative hypothesis, that ets outperforms arima, you get strong     signal that it does:

p-value = 1.774e-05
alternative hypothesis: greater

I wonder obout this result, because the aic criteria for the arima models were much lower than for ets.

I hope it helps you a little bit.

  • $\begingroup$ Thanks for the reply JohnnyB. The dm.test is a helpful method to compare forecast between model classes. And regarding the aic being lower of arima, in general AIC are not comparable between model classes as they use different no of parameters to derive that. $\endgroup$
    – priyaj
    May 24, 2013 at 4:44
  • $\begingroup$ Are you sure about the non-comparability of aic from different models? Because smaller error-variance decreases aic and the number of parameters increases it. So, if a model has lower variance because of the higher number of parameters, the number of parameters will on the other side increse the aic? It seems to me, that comparison makes sence. $\endgroup$
    – DatamineR
    May 24, 2013 at 13:22
  • 1
    $\begingroup$ See stats.stackexchange.com/a/59249/159 for why the AICs are not comparable. $\endgroup$ Jun 3, 2013 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.