# Time Series Hold Out Data not used to build model

It is my understanding that if one wants to build multiple time series models on a time series that goes from 2000 to today (2015) monthly; and one wanted to use that information to forecast 3 months in the future, it is common approach to split your data into "train" and "test" datasets.

Your test dataset would be the last 3 months of your time series (jan 2015, feb 2015, march 2015) (pretend we're already in april for simplicity sake). You would then 'define' your model on your training dataset, and then compute it's errors against your test dataset (defined as predicted vs actual).

This way you could try out many multiple models and pick the one with the lowest "forecast prediction error".

However my question is: By ignorning those last 3 months of data, how do you then use that model to forecast values later in time? Example: say you wanted to forecast April-June. Is it standard procedure to apply the same model (that wasn't built on the last 3 months) to the April - June forecast period?

If so is this something you can do in R with a package? It seems like the forecast function only works to forecast forward from the training dataset, and you can't apply it to other time series objects.

Or does one 're-build' the model on the entire time series (2000 to 2015 March) and then use that model to forecast into April-June?

I am pretty confused by this and any help would be appreciated.

Well, technically you should have your model ex ante determined, and the so-called train data set is really just to demonstrate the universal validity of your model on "the present day" (test) data set. It is a subsample test.

Once the model you're using is shown to work robustly as a predictor in all important subsamples, and you've argued convincingly that it is universally valid, it's typical to run the whole thing again using all the data and try to predict the future. This part usually tends to be the problematic part.

Here's an attempt to explain this with an example.
Say we want a function or black box
$\qquad$ In: 12 months data, say rainfall
$\qquad$ Out: a forecast of the next 3 months' rainfall
$\qquad$ E.g. Jan .. Dec --> next Jan
$\qquad \quad \ \$ Feb .. Jan --> next Feb ...

Assume further that some magic process, trained on the data from 2000 .. 2014, tells us that a good predictor of 12 months --> the next 3 months is exactly
$\qquad$ 50 % year ago $\ + \$ 50 % month ago.
E.g.

(Jan 2000 + Dec 2000) / 2  ~=  Jan 2001
...
(Dec 2013 + Nov 2014) / 2  ~=  Dec 2014


(Exercise: plot these 50 - 50 estimates against the real values in 2000 .. 2014, for your data.)

Now say the numbers for 2014 happen to be exactly 1 2 3 .. 12. Then our 50 - 50 predictor gives

Jan 2014 .. Dec 2014  1 2 3 .. 12           -->  Jan 2015 forecast (1 + 12) / 2 = 6.5
Feb 2014 .. Jan 2015    2 3 .. 12 6.5       -->  Feb 2015 forecast (2 + 6.5) / 2 = 4.25
Mar 2014 .. Feb 2015      3 ..    6.5 4.25  -->  Mar 2015 forecast (3 + 4.25) / 2 = 3.625


Do you see how the Feb 2015 forecast uses the Jan 2015 forecast 6.5, and the Mar 2015 forecast uses both the Jan 6.5 and Feb 4.25 ?
(Exercise: repeat this with numbers 1 .. 12, but 20 % year ago $\ + \$ 80 % month ago; repeat on your real data.)

Now we can compare the 3-month forecasts trained on 2000 .. 2014 with the real values for Jan Feb Mar 2015, which is the reason for all this (holding out the last 3 months). Otherwise, we'd have a black-box forecaster with magic parameters (50 - 50) but no independent test of how it performs on new data.