I’ve already written this question, but probably I didn’t specified it well, for this reason I write it again. I need to use a random walk model (no-change) yt = yt(1+t) to compute the ratio of RMSFE. What I would like to do is:

  1. Fit the model to the data yt,...,yt+k−1 and let yˆt+k be the forecast for the next observation.
  2. Compute the forecast error as et=yˆt+k−yt+k.
  3. Repeat for t=1,...,n−k"

    residuals1 <- rep(0,58)
    residuals6 <- rep(0,58)
    residuals12 <- rep(0,58)
    y1 <- t(y[,1])
    for (i in 1:58) {       
      residuals1[i] <- y1[134+i+1]-y1[134+i]    
      residuals6[i] <- y1[134+i+6]-y1[134+i] 
      residuals12[i] <- y1[134+i+12]-y1[134+i] 

Is it a correct way to compute the out.of sample forecasting errors or am I missing something? I would appreciate any suggestions. Thanks!

  • $\begingroup$ I'm confused. You're subtracting something in y1 from something in y1. Is y1 the forecast or the actual data (outside of your sample)? When you say "accuracy" I assume you're comparing forecast to actual (on non-training data, "out of sample" data)? $\endgroup$ – Wayne Feb 2 '12 at 21:16
  • $\begingroup$ Thanks. I think that in a RW model the forecast data (t+k) is given by the observation at time t, this is why I'm subtracting something in y1 from something in y1. $\endgroup$ – Frank Feb 3 '12 at 7:15
  • $\begingroup$ Sorry that I'm still confused... Forecast error is the difference between a forecast you make with your RW and the actual data. Presumably, you used the first N observations from your data to adjust (train) your RW model, holding back the last M observations to test with. Then you forecast M steps ahead with your model and compare that to the actual M observations. You don't compare your forecast to your forecast or your data to your data. Or am I misunderstanding? $\endgroup$ – Wayne Feb 3 '12 at 15:33
  • $\begingroup$ Yeah I understand your point. But since I'm doing a RW I say that my forecast (t+k) is y at time t. $\endgroup$ – Frank Feb 3 '12 at 17:02
  • $\begingroup$ Anyhow I tried to do the loop in your way, by estimating a RW model in DLM form, but it gave me the same result cause V=0. $\endgroup$ – Frank Feb 3 '12 at 17:12

As you are using a naive (random walk) model, there are no parameters, so it makes no sense to talk of in-sample and out-of-sample, or of training sets and test sets. There is nothing to train.

On the other hand, you may be doing this to compare the results with other methods applied to a training set. I'll assume that's the case.

I'm guessing your data consists of 204 observations (134+58+12) and you are using the first 135 as a training set. There's no need for loops here as the computation is easily vectorized. Also, because you use a loop you ignore some of the available forecast errors in the test set.

I think you can get the results you want as follows:

y <- rnorm(204)
residuals1 <- diff(y[135:204],lag=1)
residuals6 <- diff(y[135:204],lag=6)
residuals12 <- diff(y[135:204],lag=12)
  • $\begingroup$ Thanks. Yes what I'm trying to do is compare the RW results with other methods. I tried your specification, but I obtain the same results I had with the loop. Maybe this specification is too much naive? $\endgroup$ – Frank Feb 3 '12 at 7:14
  • $\begingroup$ You can't have the same results for 1-step and 6-step as there are different numbers of residuals doing it my way compared to the loop. $\endgroup$ – Rob Hyndman Feb 3 '12 at 10:01
  • $\begingroup$ Yes I just checked the 12 step ahead. I have less numbers of residuals, your way it's more correct and faster, but it didn't solve my problem, although I can't think any other way to express it. Thanks $\endgroup$ – Frank Feb 3 '12 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.