1
$\begingroup$

Background:

  • I'm training a Neural Network for a classification task.
  • I have a dataset consisting of 1M samples (100 features for each sample), that was collected over a period of 5 days. The data for each feature always comes from the same sensor. One example of a feature is a temperature sensor.
  • My training and validation sets are sampled (without shuffling) with 8-fold cross-validation from the data of the first 4 days. Or to phrase it differently: I always use data from 3.5 days for training, and the consecutive data samples from half a day for validation purposes. So to be clear, this half day's worth of data could be from either of the four first days.
  • My test set is the data from the 5th day.
  • My model takes one data sample at a time as an input and outputs a prediction for the correct class. No historical measurements are included. No predictions of future states are done. A data sample only ever contains the current sensor readings.

My concern: From what I understand, applying cross-validation to time series data is usually done slightly different than what I presented above (see for example this post), e.g. with a type of forward-chaining / rolling method, to avoid a look-ahead bias and because we can't fully expect completely i.i.d data samples. My data could be said to be a sort of time series, even though I do not necessarily model and treat it that way. For example, I only ever feed 1 data sample at a time to my network, without including any historical data measurements. Because of this, my gut tells me that it should be fine to use a normal k-fold cross-validation in this particular case, and that I would only need to change the approach if it was properly modelled as a time series task (for example by feeding several historical data samples at a time to my model to estimate the current state). Is my gut right or wrong about this? If it is not, why?

$\endgroup$
2
  • $\begingroup$ It is a little strange how with only one sample point you can fit something (or maybe I misunderstood something). In any case, for ts data, I would recommend to NOT resample your data in cross-validation step as this definitely destroys ts dependence. $\endgroup$ Mar 18 '20 at 22:59
  • $\begingroup$ $(1)$ Do any of your $100$ features contain, directly or indirectly, any temporal information? $(2)$ During exploratory data analysis, did the data indicate they contained any temporal dependence? If the answer to $(1)$ or $(2)$ is yes, which there is a good chance for spatially fixed sensors and physical processes, you are modeling (spatial-)time series data. Also, batch-size of one training seems like it would be excruciatingly slow for a data set of the size stated - and if your network truly does not account for “historical” measurements, how exactly is it learning? $\endgroup$ Mar 19 '20 at 9:17
1
+50
$\begingroup$

My intuition is that you should apply cross validation as if it is a time series. Although, like you said you are only predicting from individual time points, your validation accuracy will be biased due to autocorrelation in your samples.

For example, lets say you train a NN on days 0-4.5 and test on days 4.5-5. The problem here is that the last training example will likely be very similar to the first test example if there is strong autocorrelation (which there would be with temperature assuming time spacing of 5days/1million = 0.43s apart). Therefore this is a little bit like having some of the same samples in your test set as what you trained on and so you you are likely to calculate a better validation score than you would if you just had random samples in your test set. This in turn is likely to make you overtrain your network.

If you were looking for a quick way to reduce this you could cross validate by leaving a gap in time between your training set and you test set. For example lets say it takes 1hr to get rid of the autocorrelation. You could then train on hours 1-107.999 (days 1-4.5) and test on hours 109-119.999.

If you wanted to test on a fold in the middle of the sequence you could leave an hour gap on both sides.

And to evaluate how long you should leave as a gap you could test this. Like train a network and plot the validation score vs gap left. I would expect this curve to tend towards a lower and more reliable value as the gap is increased.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.