4
$\begingroup$

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use.

In any case, I've been doing some reading on the two most common methods, hold out sample (use a training sample to predict the last N observations in the test sample) ---- and cross-validation/ rolling-origin forecast/ K-1 validation, however it's named.

I'm not sure exactly what the latter means when referring to a time series. I guess that means take all data up to right before a certain time X, then forecast X, for basically every X on the timeline, and average that accuracy. Obviously for seasonal data you would be starting at a point 1 or 2 seasons along the timeline.

Apparently this second method may have issues because the earlier points in time are used more often? Or something of that sort.

Anyway -- I'm just looking for one simple, effective way to measure an ETS or Arima time-series forecast, using an out-of-sample method, which I think most represents future unknown data. Is there a simple way to perform this in R?

I've tried the accuracy() method, but I'm failing to understand exactly what objects to pass into it's arguments. "F" is the forecast. I'm a bit confused whether to pass the model object created by ets() or arima() or a forecast object created by forecast() of these models. The forecast object contains FUTURE points from the data -- don't we need mock forecasts for prior points for an out-of-sample test?

A bit more confusing is the "x" parameter (defined as the actual values) that is needed for an out-of-sample test. I haven't been able to provide an object of a suitable format that doesn't result in an error. I guess it must be a vector of actual numbers as the same length as the model? What is the best way to import a list as a vector? I'm just a bit confused here. Thanks!

$\endgroup$
  • $\begingroup$ Commenting on my own question -- maybe I figured this out (at least the first method, the hold out sample of the last N months). I divided my data between "train" and "test" sections, and then followed this advice to create "test" forecasts: robjhyndman.com/hyndsight/out-of-sample-one-step-forecasts library(forecast) fit <- ets(trainingdata) fit2 <- ets(testdata, model=fit) onestep <- fitted(fit2) I then did accuracy (onestep, testdata) ---- not sure if this is correct. $\endgroup$ – John Babson Nov 19 '14 at 21:07
  • $\begingroup$ Alright that last comment I may have been deeply confused --- I divided my data between training and test sets. I created an ETS model from the training data. Then I forecasted the training data out for 7 months using forecast(training.ets, h=7), the exact length of the test data. Then I did accuracy (forecast, test_data) -- it appeared to accept the input finally. Does that look right? I'm still confused how to do other methods of cross-validation, like the K-1 cross validation. $\endgroup$ – John Babson Nov 19 '14 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.