Cross-validation with time series data, when dealing with one data sample at a time?

Question

Background:

• I'm training a Neural Network for a classification task.
• I have a dataset consisting of 1M samples (100 features for each sample), that was collected over a period of 5 days. The data for each feature always comes from the same sensor. One example of a feature is a temperature sensor.
• My training and validation sets are sampled (without shuffling) with 8-fold cross-validation from the data of the first 4 days. Or to phrase it differently: I always use data from 3.5 days for training, and the consecutive data samples from half a day for validation purposes. So to be clear, this half day's worth of data could be from either of the four first days.
• My test set is the data from the 5th day.
• My model takes one data sample at a time as an input and outputs a prediction for the correct class. No historical measurements are included. No predictions of future states are done. A data sample only ever contains the current sensor readings.

My concern: From what I understand, applying cross-validation to time series data is usually done slightly different than what I presented above (see for example this post), e.g. with a type of forward-chaining / rolling method, to avoid a look-ahead bias and because we can't fully expect completely i.i.d data samples. My data could be said to be a sort of time series, even though I do not necessarily model and treat it that way. For example, I only ever feed 1 data sample at a time to my network, without including any historical data measurements. Because of this, my gut tells me that it should be fine to use a normal k-fold cross-validation in this particular case, and that I would only need to change the approach if it was properly modelled as a time series task (for example by feeding several historical data samples at a time to my model to estimate the current state). Is my gut right or wrong about this? If it is not, why?

Answer 1

My intuition is that you should apply cross validation as if it is a time series. Although, like you said you are only predicting from individual time points, your validation accuracy will be biased due to autocorrelation in your samples.

For example, lets say you train a NN on days 0-4.5 and test on days 4.5-5. The problem here is that the last training example will likely be very similar to the first test example if there is strong autocorrelation (which there would be with temperature assuming time spacing of 5days/1million = 0.43s apart). Therefore this is a little bit like having some of the same samples in your test set as what you trained on and so you you are likely to calculate a better validation score than you would if you just had random samples in your test set. This in turn is likely to make you overtrain your network.

If you were looking for a quick way to reduce this you could cross validate by leaving a gap in time between your training set and you test set. For example lets say it takes 1hr to get rid of the autocorrelation. You could then train on hours 1-107.999 (days 1-4.5) and test on hours 109-119.999.

If you wanted to test on a fold in the middle of the sequence you could leave an hour gap on both sides.

And to evaluate how long you should leave as a gap you could test this. Like train a network and plot the validation score vs gap left. I would expect this curve to tend towards a lower and more reliable value as the gap is increased.