Is it better to split sequences into overlapping or non-overlapping training samples?

Question

I have $N$ (time) sequences of data with length $2048$. Each of these sequences correseponds to a different target output. However, I know that only a small part of the sequence is needed to predict this target output, say a sub-sequence of length $128$.

I could split up each of the sequences into $16$ partitions of $128$, so that I end up with $16N$ training smaples. However, I could drastically increase the number of training samples if I use a sliding window instead: there are $2048-128 = 1920$ unique sub-sequences of length $128$ that preserve the time series. That means I could in fact generate $1920N$ unique training samples, even though most of the input is overlapping.

I could also use a larger increment between individual "windows", which would reduce the number of sub-sequences but it could remove any autocorrelation between them.

Is it better to split my data into $16N$ non-overlapping sub-sequences or $1920N$ partially overlapping sub-sequences?

Answer 1

You are correct to question the dependence between succecssice slices of training data as indeed they are not fully independent samples. That said, I suggest using these partialy overallaping sub-sequences, i.e. the widely-used sliding-window approach when training. It is very common for (D)NNs; Keras even has a pre-defined function to do exactly that timeseries_dataset_from_array.

There are not hard absolute rules on the choice of window sizes or their overlap. For example, even in simple ARIMA application it is unclear if a fixed size window or a fixed origin window is always better than another. What is though very likely is that using non-overlapping sequences will not provide enough samples to train experessive models as DNNs (LSTM?). If we are very worried about leakage we might use "purging" (not a commonly used term in my opinion). In effect we create a buffer between our training and test data. That way while training there is "some overlap" between samples' training features and the response, when testing no time-point $t_i$ from training appearing in the test set (see Lopez de Prado (2018) "Advances in Financial Machine Learning" Chapt. 7 Cross-validation in Finance for more details). I have not seen this approach being widely used but it might be worth exploring. In any case, I would urge checking, after establishing some notion of stationarity, the ACF/PACF plot of the series to get an idea of what potential time-lagged correlations we might expect.

Finally, I think it is worth emphasing that it preprocessing the time-series data at hand is rather relevant. Especially if the time-series exhibit strong seasonlity, although a DNN should be able to deal with them well, it often helps to remove the seasonality before applying a DNN procedure.