3
$\begingroup$

I have $N$ (time) sequences of data with length $2048$. Each of these sequences correseponds to a different target output. However, I know that only a small part of the sequence is needed to predict this target output, say a sub-sequence of length $128$.

I could split up each of the sequences into $16$ partitions of $128$, so that I end up with $16N$ training smaples. However, I could drastically increase the number of training samples if I use a sliding window instead: there are $2048-128 = 1920$ unique sub-sequences of length $128$ that preserve the time series. That means I could in fact generate $1920N$ unique training samples, even though most of the input is overlapping.

I could also use a larger increment between individual "windows", which would reduce the number of sub-sequences but it could remove any autocorrelation between them.

Is it better to split my data into $16N$ non-overlapping sub-sequences or $1920N$ partially overlapping sub-sequences?

$\endgroup$
2
  • 1
    $\begingroup$ I'd suspect that there is quite some dependence within the same sequence. This means that whatever way you use subsequences from the same sequence, this will not give you independent training data, so their number will not really reflect the amount of training information you have generated, and it seems to me that this dependence somehow needs to be modelled in order to make good use of different subsamples from the same sequence. Ultimately this depends on the nature of your data. $\endgroup$ Oct 15 '20 at 19:54
  • 1
    $\begingroup$ Overlapping will increase dependence even more, but avoiding it is not enough to have independence. The idea to have longer breaks to reduce autocorrelation is not bad but your description doesn't look like you can ever be sure that no correlation or dependence is left, so ultimately you have to deal with dependence whatever you do, unless you only use one training sample per sequence (assuming that at least different sequences are independent). $\endgroup$ Oct 15 '20 at 20:00
3
+50
$\begingroup$

You are correct to question the dependence between succecssice slices of training data as indeed they are not fully independent samples. That said, I suggest using these partialy overallaping sub-sequences, i.e. the widely-used sliding-window approach when training. It is very common for (D)NNs; Keras even has a pre-defined function to do exactly that timeseries_dataset_from_array.

There are not hard absolute rules on the choice of window sizes or their overlap. For example, even in simple ARIMA application it is unclear if a fixed size window or a fixed origin window is always better than another. What is though very likely is that using non-overlapping sequences will not provide enough samples to train experessive models as DNNs (LSTM?). If we are very worried about leakage we might use "purging" (not a commonly used term in my opinion). In effect we create a buffer between our training and test data. That way while training there is "some overlap" between samples' training features and the response, when testing no time-point $t_i$ from training appearing in the test set (see Lopez de Prado (2018) "Advances in Financial Machine Learning" Chapt. 7 Cross-validation in Finance for more details). I have not seen this approach being widely used but it might be worth exploring. In any case, I would urge checking, after establishing some notion of stationarity, the ACF/PACF plot of the series to get an idea of what potential time-lagged correlations we might expect.

Finally, I think it is worth emphasing that it preprocessing the time-series data at hand is rather relevant. Especially if the time-series exhibit strong seasonlity, although a DNN should be able to deal with them well, it often helps to remove the seasonality before applying a DNN procedure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.