# Using rolling windows to compute out of sample accuracy

I’ve already written this question, but probably I didn’t specified it well, for this reason I write it again. I need to use a random walk model (no-change) yt = yt(1+t) to compute the ratio of RMSFE. What I would like to do is:

1. Fit the model to the data yt,...,yt+k−1 and let yˆt+k be the forecast for the next observation.
2. Compute the forecast error as et=yˆt+k−yt+k.
3. Repeat for t=1,...,n−k"

residuals1 <- rep(0,58)
residuals6 <- rep(0,58)
residuals12 <- rep(0,58)

y1 <- t(y[,1])
for (i in 1:58) {
residuals1[i] <- y1[134+i+1]-y1[134+i]
residuals6[i] <- y1[134+i+6]-y1[134+i]
residuals12[i] <- y1[134+i+12]-y1[134+i]
}


Is it a correct way to compute the out.of sample forecasting errors or am I missing something? I would appreciate any suggestions. Thanks!

-
show 1 more comment

As you are using a naive (random walk) model, there are no parameters, so it makes no sense to talk of in-sample and out-of-sample, or of training sets and test sets. There is nothing to train.

On the other hand, you may be doing this to compare the results with other methods applied to a training set. I'll assume that's the case.

I'm guessing your data consists of 204 observations (134+58+12) and you are using the first 135 as a training set. There's no need for loops here as the computation is easily vectorized. Also, because you use a loop you ignore some of the available forecast errors in the test set.

I think you can get the results you want as follows:

y <- rnorm(204)
residuals1 <- diff(y[135:204],lag=1)
residuals6 <- diff(y[135:204],lag=6)
residuals12 <- diff(y[135:204],lag=12)

-
 Thanks. Yes what I'm trying to do is compare the RW results with other methods. I tried your specification, but I obtain the same results I had with the loop. Maybe this specification is too much naive? – Frank Feb 3 '12 at 7:14 You can't have the same results for 1-step and 6-step as there are different numbers of residuals doing it my way compared to the loop. – Rob Hyndman Feb 3 '12 at 10:01 Yes I just checked the 12 step ahead. I have less numbers of residuals, your way it's more correct and faster, but it didn't solve my problem, although I can't think any other way to express it. Thanks – Frank Feb 3 '12 at 16:41