How to deal with the problem of correlated samples from time series in machine learning

Question

We know that most of the machine learning algorithms assume the samples are i.i.d. However if the samples are from time series (like ARMA in the prediction of time series), then they are highly correlated. Then how to deal with such correlated samples? Note that, here I mean the correlation between samples not the correlation between features.

1. Example 1: we have a time series $\{x_1,\cdots,x_T\}.$ Then we use three times to predict the forth time to construct the training samples:

$$(X_1 = (x_1,x_2,x_3), y_1 = x_4), (X_2 = (x_2,x_3,x_4), y_2 = x_5), \cdots.$$ Then the training samples $(X_i,y_i)$ must be highly correlated, which is contradict to the assumption of i.i.d. samples for most machine learning algorithms.

2. Example 2: we have a time series $\{x_1,\cdots,x_T\}$ and the corresponding label series $\{y_1,\cdots,y_T\}.$ Then the training samples:

$$(x_1, y_1), (x_2, y_2), \cdots$$

is also highly correlated.

I have the following guess:

1. In principle, machine learning algorithm is not a proper way to deal with the time series?

2. The assumption of i.i.d. samples is actually not necessary for the machine learning algorithm?

I think it is very basic questions. Can anyone give me some references?

Answer 1

Then how to deal with such correlated samples?

This boils down to how these systems are trained, i.e., sequence-to-sequence training. If we do that, the correlation structure is preserved, such as LSTMs or any RNN's training.

machine learning algorithm is not a proper way to deal with the time series?

Correlations are encoded in the models such as Gaussian process regression requires covariance matrix i.e., Kernel matrices, these matrices encodes training sample's correlation structure. See Usual covariance functions.

The assumption of i.i.d. samples is actually not necessary for the machine learning algorithm?

Time-series modelling may not be classified as "vanilla supervised learning". So from statistical learning theory perspective, they are special class of models with added assumptions. So, how time-series model's set up will not justify the argument of i.i.d. samples is actually not necessary for the machine learning algorithm.

Answer 2

There are plenty of machine learning models that are all about possibly correlated data. Examples include LSTM and transformer neural networks, Gaussian processes, AR/MA/ARMA/ARIMA/etc. models, various forms of mixed models such as hierarchical models, spatial models, GAMMs and so on. Ignoring an existing correlation is often a very bad idea and using a model that can reflect a correlation is usually a good idea. There's also various techniques to try to reduce the time series nature of data (e.g. building a lot of features based on the past history etc.) that might make ML models that assume i.i.d. observations fit better.

One way to evaluate whether using a model that ignores correlations is an option is to use some suitable form of (cross-)validation such as past-vs.-future splits for time series or sampling clusters (rather than individual records) in cross-validation in case of clustered data. If with a good validation strategy a model that ignores the correlations works great, then maybe it's an option.