15
$\begingroup$

What's the best way to split time series data into train/test/validation sets, where the validation set would be used for hyperparameter tuning?

We have 3 years' worth of daily sales data, and our plan is to use 2015-2016 as the training data, then randomly sample 10 weeks from the 2017 data to be used as the validation set, and another 10 weeks from 2017 data for the test set. We'll then do a walk forward on each of the days in the test and validation set.

Improve this question
$\endgroup$
11
$\begingroup$

You should use a split based on time to avoid the look-ahead bias. Train/validation/test in this order by time.

The test set should be the most recent part of data. You need to simulate a situation in a production environment, where after training a model you evaluate data coming after the time of creation of the model. The random sampling you use for validation and training is therefore not good idea.

Improve this answer
$\endgroup$
11
$\begingroup$

I think the most complete way to leverage your time-series data for training/validation/testing/prediction is this:

enter image description here

Is the picture self explanatory? If not, please comment and I will add more text...

Improve this answer
$\endgroup$
4
  • $\begingroup$ I found your chart very useful, as I'm in trouble with the same topic. 1.Not sure I got the second step of the chart. What do the V and T squares mean? V should not always be later than T? 2.are we saying that, after best model selection, for tomorrow's forecast, we use all data and get the right parameters using the best model, but using the full data-set ? Thanks! Gioele $\endgroup$
    – Gioele
    Feb 18 at 14:17
  • 2
    $\begingroup$ Where does this picture come from ? $\endgroup$
    – manu190466
    Feb 18 at 14:42
  • 1
    $\begingroup$ @Gioele 1. T=Training samples V=Validation samples. 2. yes, you try to reuse the historical data as much as possible, first to select the best model and then to fit the best model. $\endgroup$
    – elemolotiv
    Feb 19 at 22:30
  • 2
    $\begingroup$ @manu190466 it's my personal attempt to combine available approaches: cross-validation to select the best model today, backtest to fit the selected model, walkforward analysis to check if the whole idea would have worked in history, day after day until today $\endgroup$
    – elemolotiv
    Feb 19 at 22:33
4
$\begingroup$

Instead of creating only one set of training/validation set, you could create more such sets.

The first training set could be, say, 6 months data (first semester of 2015) and the validation set would then be the next three months (July-Aug 2015). The second training set would be a combination of the first training and validation set. The validation set is then the next three months (Sept-Oct 2015). And so on.

This is a variation of K-Fold cross-validation where the training sets are a combination of the previous training and validation set.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Downvote. The training set is split "k-fold" into training and validation set (T&V in the image of the other answer), no need to put the validation set at the end of time, since then, you always lose the most recent months for training the model and sacrifice them just to get the best validation during training. This will make the model training set "outdated" in relation to the testing set's evaluations. Validation should accompany the training set and not create a new full split, therefore "cross-validation". You recommend an uncrossed "end-of-training-time validation", not recommended! $\endgroup$
    – questionto42
    May 22 at 20:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.