Forecast accuracy measures for different forecast horizon h in R

Question

I have a yearly time series data, from 1980 to 2005. The data is splitted into an training sample and a out of sample; the out-of sample consists of the 6 most recent observations and the rest is considered for training sample. I need to fit a ETS model and compare different accuracy measures for different forecast step aheads h=1,2,3,4,5 and 6.

Something like this:

      h=1  h=2  h=3  h=4  h=5  h=6  ...
MSE   ..   ..   ..   ..   ..   ..   ...
MASE  ..   ..   ..   ..   ..   ..   ...

The following code gives me the accuracy measures for h=6:

trainx<- window(x,end=1999.99)
testx<- window(x,start=2000)
fit<- ets(trainx)
accuracy(forecast(fit,h=6),testx)

The questions are:

1. How can I calculate the accuracy measures for h=1,2,3,4,5 ? For instance, when h=2, I fit a model to training data and I produce the forecast that correspond to 2000 and 2001.

2. Now, how should I produce the forecast for 2002 and 2003, etc?
   Should I suppose that the observations for the year 2000 an 2001 are known and then fit a new model (this time I need to add the observations of 2000 and 2001 to the training set), then, to produce the forecast for 2002 and 2003?

Answer 1

It sounds like you need a rolling forecast origin (aka time series cross-validation). Here is an example.

library(forecast)
x <- ts(cumsum(rnorm(26)), start=1980)

k <- 10 # minimum data length for fitting a model
n <- length(x)
mae <- matrix(NA,n-k-1,6)
st <- tsp(x)[1] + k - 1

for(i in 1:(n-k-1))
{
  trainx <- window(x, end=st+i-1)
  testx <- window(x, start=st+i, end=st+i+5)
  fit <- ets(trainx)
  fcast <- forecast(fit, h=6)
  mae[i,1:length(testx)] <- abs(fcast[['mean']]-testx)
}
mase <- mae / mean(abs(diff(x)))
tab <- rbind(colMeans(mae,na.rm=TRUE),colMeans(mase,na.rm=TRUE))
rownames(tab) <- c("MAE","MASE")
colnames(tab) <- paste("h=",1:6,sep="")
tab