3
$\begingroup$

I am using forecast() package in R to predict future values. I have a time series data for approx 6-7 years.

First, I split the data into training set and test set. Test set contained values of the last 12 months of original data. Then, applied ets() and auto.arima() on the training set. This can be done by iterating 12 times using 1-step ahead OR by running once to get 12 points at the same time, which is my question no. 1

Then, the output received is compared with the test set to check which of the two (ets or auto.arima) gives minimal error in comparison with test set. The selected model class is then applied on the original data set to predict future 12 values. So, here arises my question no. 2

In Short, I have two things to do

  1. Compare accuracy measures for test set of ets and auto.arima to select one
  2. Predict future values using the derived model from the first step

Questions

  1. Should I use one-step ahead or 12-step ahead forecasts to generate data for test set?

  2. Should I use one-step ahead or 12-step ahead forecasts to predict future values?

  3. Should the method be same in both the steps mentioned above?

Kindly help.

$\endgroup$
  • 1
    $\begingroup$ What advice you are seeking here is unclear. It is not clear whether you have already selected the better method or need to do that: different parts of your post appear contradictory. 1-step and 12-step (and others too) could both be interesting and useful. Comparing different methods is usually a good idea, whether they produce similar or different results. It doesn't make much sense to compare 1-step from one method with 12-step from another; I imagine most readers would expect both. In short, I don't think you give enough information to enable us to advise much on your decisions. $\endgroup$ – Nick Cox May 23 '13 at 10:42
  • $\begingroup$ made changes to the question and explained in more detail now..hope this helps $\endgroup$ – priyaj May 23 '13 at 10:56
  • 1
    $\begingroup$ This is now for time series experts. I am not one, but I see no reason to revise my comments. $\endgroup$ – Nick Cox May 23 '13 at 11:02
2
$\begingroup$

Your questions are still not very clear, but if you want to compare the forecast performances of the two methods maybe you can do it like this?

library(forecast)
data<-USAccDeaths
tr.data<-window(data,statrt=c(1973,1),end=c(1977,12)) # Training data
test.data<-window(data,statrt=c(1978,1),end=c(1978,12)) # Test data

Models:

arima_model<-auto.arima(tr.data)
ets_model<-ets(tr.data)

Forcasts with the models:

arima_f<-forecast(arima_model,h=12)
ets_f<-forecast(ets_model,h=12)

Extracting the 12-step-ahead forecast errors from the models:

arima_f_e<-test.data-arima_f$mean
    ets_f_e<-test.data-ets_f$mean

Calculation of MSE for the forecasts:

mean((ets_f_e<-test.data-ets_f$mean)^2)
72275.59 => MSE of ets-forecast

mean((arima_f_e<-test.data-arima_f$mean)^2) 
83421.72 => MSE of arima-forecast


Application of dm.test from forecast-package to compare predictive accuracy:

dm.test(arima_f_e,ets_f_e)
=>p-value = 0.6145 
although  the MSE of ets is smaller, hypothes of equal performance cannot be rejected for the 12-step forecasts

You can also compare the errors within the training period:

dm.test(arima_model$res,ets_model$res) 
p-value = 3.548e-05 
=> the hypothesis of equal performance of one-step forecasts is rejected!

If you modify the alternative hypothesis, that ets outperforms arima, you get strong     signal that it does:

dm.test(arima_model$res,ets_model$res,alternative="greater")
p-value = 1.774e-05
alternative hypothesis: greater

I wonder obout this result, because the aic criteria for the arima models were much lower than for ets.

I hope it helps you a little bit.

$\endgroup$
  • $\begingroup$ Thanks for the reply JohnnyB. The dm.test is a helpful method to compare forecast between model classes. And regarding the aic being lower of arima, in general AIC are not comparable between model classes as they use different no of parameters to derive that. $\endgroup$ – priyaj May 24 '13 at 4:44
  • $\begingroup$ Are you sure about the non-comparability of aic from different models? Because smaller error-variance decreases aic and the number of parameters increases it. So, if a model has lower variance because of the higher number of parameters, the number of parameters will on the other side increse the aic? It seems to me, that comparison makes sence. $\endgroup$ – DatamineR May 24 '13 at 13:22
  • 1
    $\begingroup$ See stats.stackexchange.com/a/59249/159 for why the AICs are not comparable. $\endgroup$ – Rob Hyndman Jun 3 '13 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.