Using rolling windows to compute out of sample accuracy

I’ve already written this question, but probably I didn’t specified it well, for this reason I write it again. I need to use a random walk model (no-change) yt = yt(1+t) to compute the ratio of RMSFE. What I would like to do is:

1. Fit the model to the data yt,...,yt+k−1 and let yˆt+k be the forecast for the next observation.
2. Compute the forecast error as et=yˆt+k−yt+k.
3. Repeat for t=1,...,n−k"

residuals1 <- rep(0,58)
residuals6 <- rep(0,58)
residuals12 <- rep(0,58)

y1 <- t(y[,1])
for (i in 1:58) {
residuals1[i] <- y1[134+i+1]-y1[134+i]
residuals6[i] <- y1[134+i+6]-y1[134+i]
residuals12[i] <- y1[134+i+12]-y1[134+i]
}

Is it a correct way to compute the out.of sample forecasting errors or am I missing something? I would appreciate any suggestions. Thanks!

• I'm confused. You're subtracting something in y1 from something in y1. Is y1 the forecast or the actual data (outside of your sample)? When you say "accuracy" I assume you're comparing forecast to actual (on non-training data, "out of sample" data)? – Wayne Feb 2 '12 at 21:16
• Thanks. I think that in a RW model the forecast data (t+k) is given by the observation at time t, this is why I'm subtracting something in y1 from something in y1. – Frank Feb 3 '12 at 7:15
• Sorry that I'm still confused... Forecast error is the difference between a forecast you make with your RW and the actual data. Presumably, you used the first N observations from your data to adjust (train) your RW model, holding back the last M observations to test with. Then you forecast M steps ahead with your model and compare that to the actual M observations. You don't compare your forecast to your forecast or your data to your data. Or am I misunderstanding? – Wayne Feb 3 '12 at 15:33
• Yeah I understand your point. But since I'm doing a RW I say that my forecast (t+k) is y at time t. – Frank Feb 3 '12 at 17:02
• Anyhow I tried to do the loop in your way, by estimating a RW model in DLM form, but it gave me the same result cause V=0. – Frank Feb 3 '12 at 17:12

As you are using a naive (random walk) model, there are no parameters, so it makes no sense to talk of in-sample and out-of-sample, or of training sets and test sets. There is nothing to train.

On the other hand, you may be doing this to compare the results with other methods applied to a training set. I'll assume that's the case.

I'm guessing your data consists of 204 observations (134+58+12) and you are using the first 135 as a training set. There's no need for loops here as the computation is easily vectorized. Also, because you use a loop you ignore some of the available forecast errors in the test set.

I think you can get the results you want as follows:

y <- rnorm(204)
residuals1 <- diff(y[135:204],lag=1)
residuals6 <- diff(y[135:204],lag=6)
residuals12 <- diff(y[135:204],lag=12)
• Thanks. Yes what I'm trying to do is compare the RW results with other methods. I tried your specification, but I obtain the same results I had with the loop. Maybe this specification is too much naive? – Frank Feb 3 '12 at 7:14
• You can't have the same results for 1-step and 6-step as there are different numbers of residuals doing it my way compared to the loop. – Rob Hyndman Feb 3 '12 at 10:01
• Yes I just checked the 12 step ahead. I have less numbers of residuals, your way it's more correct and faster, but it didn't solve my problem, although I can't think any other way to express it. Thanks – Frank Feb 3 '12 at 16:41