I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use.
In any case, I've been doing some reading on the two most common methods, hold out sample (use a training sample to predict the last N observations in the test sample) ---- and cross-validation/ rolling-origin forecast/ K-1 validation, however it's named.
I'm not sure exactly what the latter means when referring to a time series. I guess that means take all data up to right before a certain time X, then forecast X, for basically every X on the timeline, and average that accuracy. Obviously for seasonal data you would be starting at a point 1 or 2 seasons along the timeline.
Apparently this second method may have issues because the earlier points in time are used more often? Or something of that sort.
Anyway -- I'm just looking for one simple, effective way to measure an ETS or Arima time-series forecast, using an out-of-sample method, which I think most represents future unknown data. Is there a simple way to perform this in R?
I've tried the accuracy() method, but I'm failing to understand exactly what objects to pass into it's arguments. "F" is the forecast. I'm a bit confused whether to pass the model object created by ets() or arima() or a forecast object created by forecast() of these models. The forecast object contains FUTURE points from the data -- don't we need mock forecasts for prior points for an out-of-sample test?
A bit more confusing is the "x" parameter (defined as the actual values) that is needed for an out-of-sample test. I haven't been able to provide an object of a suitable format that doesn't result in an error. I guess it must be a vector of actual numbers as the same length as the model? What is the best way to import a list as a vector? I'm just a bit confused here. Thanks!
