What's the best way to split time series data into train/test/validation sets, where the validation set would be used for hyperparameter tuning?
We have 3 years' worth of daily sales data, and our plan is to use 2015-2016 as the training data, then randomly sample 10 weeks from the 2017 data to be used as the validation set, and another 10 weeks from 2017 data for the test set. We'll then do a walk forward on each of the days in the test and validation set.
You should use a split based on time to avoid the look-ahead bias. Train/validation/test in this order by time.
The test set should be the most recent part of data. You need to simulate a situation in a production environment, where after training a model you evaluate data coming after the time of creation of the model. The random sampling you use for validation and training is therefore not good idea.
I think the most complete way to leverage your time-series data for training/validation/testing/prediction is this:
Is the picture self explanatory? If not, please comment and I will add more text...
In the following, "validation set" was replaced with "testing set" to be aligned with the naming in this Q/A.
Instead of creating only one set of a training/testing set, you could create more of such sets.
The first training set could be, say, 6 months data (first semester of 2015) and the testing set would then be the next three months (July-Aug 2015). The second training set would be a combination of the first training and testing set. The testing set is then the next three months (Sept-Oct 2015). And so on.
This walk over time is a bit like the k-fold cross-validation where the training sets are a combination of the previous training and validation set, and that put together. Only that in this model, this happens by walking through time to check more than one prediction, and in this model, the training and testing set are put together, not the training and validation set. You have validation by default, in the model, if you walk through time with more than one prediction. The metrics of these predictions can be put against each other.
This is the walk-forward model, see a comment below. The model image mixes up testing set with validation set. Normally, this naming issue does not matter, but here, it does, since it stands against the naming of the rest. If you then allow to make an additional k-fold validation from the testing set, you have the three sets that the question asks for. And yes, you do not need that validation set if you make a "walk-forward" with enough steps.
Thus, even if this model needs only training and testing set, it can still be an answer to the question that asks for the three sets. Since the validation set can be seen as being replaced by the "walk-forward". And it still also allows small k-fold validation as splits from the testing set, so that you could see it as 3+1 sets in the end.
