Data leakage when using walk forward optimization

Question

I am setting up a neural network that will predict the incoming customers at a store for the next seven days (the output is a list with seven numbers, one for each day). As input, I will give the neural network the last x weeks of data. The data contains the number of visitors for each day and information about that date, like what day it was (Monday, Tuesday, etc.). The optimal x has yet to be determined.

I split the data in a training- (80%) and testset (20%).

For training the neural network I am now using walk forward optimization where I take the first x weeks of the trainingset as input and use the next seven days after that as output. This is one input-output pair. For the next pair, I move the window one day. The input loses the first day, but gains the first day of the output of the previous pair. The output gains a whole new day. I hope the picture clarifies what I am talking about.

.

I wonder if this method will cause data-leakage, since I am giving the neural network output that it has already seen, one input-output pair before. I guess this would be solved by shifting the pairs by seven days instead of one, but this leaves me with very little data to train on. Keep in mind that I use this method for training the network. When testing the method, I am using data that the neural network has never seen before.

Answer 1

Your approach seems perfectly valid. Since you don't train your model on future data reserved for testing you avoid the problem of look-ahead bias.

You don't have to shift your training window forward by the seven days covered by your test period because even when you walk forward by only one day you are not using any future observations for training and constructing the forecast (apart from deterministic features such as day of week etc.).

A nice description of this procedure (also known as time series cross-validation) is given in the blog post Cross-validation for time series by Robert Hyndman.

Answer 2

From my perspective, it depends.

If you use what the diagram calls "testing set" for hyperparameter tuning, there is a risk of data leakage.

If you use what the diagram calls "testing set" for hyperparameter tuning, I'd make predictions and evaluate results on data not used for training and not used for hyperparameter tuning.

Also, be sure the dates of your "training set" target variable do not overlap with the dates of your "testing set". You might need a gap of 7 days between the "training set" and the "testing set".

If you share your code, we can better comment on what you are doing.