What's the best way to split time series data into train/test/validation sets, where the validation set would be used for hyperparameter tuning?
We have 3 years' worth of daily sales data, and our plan is to use 2015-2016 as the training data, then randomly sample 10 weeks from the 2017 data to be used as the validation set, and another 10 weeks from 2017 data for the test set. We'll then do a walk forward on each of the days in the test and validation set.
You should use a split based on time to avoid the look-ahead bias. Train/validation/test in this order by time.
The test set should be the most recent part of data. You need to simulate a situation in a production environment, where after training a model you evaluate data coming after the time of creation of the model. The random sampling you use for validation and training is therefore not good idea.
Instead of creating only one set of training/validation set, you could create more such sets.
The first training set could be, say, 6 months data (first semester of 2015) and the validation set would then be the next three months (July-Aug 2015). The second training set would be a combination of the first training and validation set. The validation set is then the next three months (Sept-Oct 2015). And so on.
This is a variation of K-Fold cross-validation where the training sets are a combination of the previous training and validation set.
I think the most complete way to leverage your time-series data for training/validation/testing/prediction is this:
Is the picture self explanatory? If not, please comment and I will add more text...
