# How to deal with the problem of correlated samples from time series in machine learning

We know that most of the machine learning algorithms assume the samples are i.i.d. However if the samples are from time series (like ARMA in the prediction of time series), then they are highly correlated. Then how to deal with such correlated samples? Note that, here I mean the correlation between samples not the correlation between features.

1. Example 1: we have a time series $$\{x_1,\cdots,x_T\}.$$ Then we use three times to predict the forth time to construct the training samples:

$$(X_1 = (x_1,x_2,x_3), y_1 = x_4), (X_2 = (x_2,x_3,x_4), y_2 = x_5), \cdots.$$ Then the training samples $$(X_i,y_i)$$ must be highly correlated, which is contradict to the assumption of i.i.d. samples for most machine learning algorithms.

1. Example 2: we have a time series $$\{x_1,\cdots,x_T\}$$ and the corresponding label series $$\{y_1,\cdots,y_T\}.$$ Then the training samples:

$$(x_1, y_1), (x_2, y_2), \cdots$$

is also highly correlated.

I have the following guess:

1. In principle, machine learning algorithm is not a proper way to deal with the time series?

2. The assumption of i.i.d. samples is actually not necessary for the machine learning algorithm?

I think it is very basic questions. Can anyone give me some references?

• I'm not sure from your question what is exactly the input. Is it only a single sample of feature set or you expect to inference on a signal of sets. For the former, since you don't have the feature signal I think it is ok to learn as is. For the latter, you can look at a signal's section (like transformers) or LSTMs for example. Jun 24 '21 at 6:25
• @Cherny pls see my update. I made two examples of training samples based on the time series which are both contradict to the assumption of i.i.d. samples for most machine learning algorithms. Jun 24 '21 at 15:36

Then how to deal with such correlated samples?

This boils down to how these systems are trained, i.e., sequence-to-sequence training. If we do that, the correlation structure is preserved, such as LSTMs or any RNN's training.

machine learning algorithm is not a proper way to deal with the time series?

Correlations are encoded in the models such as Gaussian process regression requires covariance matrix i.e., Kernel matrices, these matrices encodes training sample's correlation structure. See Usual covariance functions.

The assumption of i.i.d. samples is actually not necessary for the machine learning algorithm?

Time-series modelling may not be classified as "vanilla supervised learning". So from statistical learning theory perspective, they are special class of models with added assumptions. So, how time-series model's set up will not justify the argument of i.i.d. samples is actually not necessary for the machine learning algorithm.

• could you refer some books or papers for how machine learning or deep learning deal with the time series samples. I want to have a systematic learning on this part. Jun 26 '21 at 13:38
• Andrew Ng's course could be a nice starting point coursera.org/learn/nlp-sequence-models Jun 26 '21 at 22:38
• Thanks, as I asked in Björn's answer, if there is no i.i.d. sample assumption, maximal likelihood, central limit theorem, law of large numbers e.t.c all fail, which seems end of world, could I ask for your opinion? Jun 27 '21 at 5:33
• iid sample assumption can be rephrased for time-series, i.e., addition of for example Markovian assumption, so other concepts like CLT can be recovered. Jun 27 '21 at 6:39

There are plenty of machine learning models that are all about possibly correlated data. Examples include LSTM and transformer neural networks, Gaussian processes, AR/MA/ARMA/ARIMA/etc. models, various forms of mixed models such as hierarchical models, spatial models, GAMMs and so on. Ignoring an existing correlation is often a very bad idea and using a model that can reflect a correlation is usually a good idea. There's also various techniques to try to reduce the time series nature of data (e.g. building a lot of features based on the past history etc.) that might make ML models that assume i.i.d. observations fit better.

One way to evaluate whether using a model that ignores correlations is an option is to use some suitable form of (cross-)validation such as past-vs.-future splits for time series or sampling clusters (rather than individual records) in cross-validation in case of clustered data. If with a good validation strategy a model that ignores the correlations works great, then maybe it's an option.

• One way to evaluate whether using a model that ignores correlations, do you have some references? I want to know more details about this part. Jun 24 '21 at 16:31
• and if there is no i.i.d. sample assumption, maximal likelihood, central limit theorem, law of large numbers e.t.c all fail, which seems end of world. Jun 24 '21 at 17:04
• This is done often in Kaggle competitions & seem to work. See e.g. the winning solution of the M5-Accuracy competition, which was about predicting sales of items on each day. My intuition for why it works is that by good features you capture the time series that came before and deal with correlated records by finding regularizing hyperparameters that avoid overfitting/memorization of records via a suitable validation approach. One good discussion of validation sets is here. Jun 24 '21 at 22:52
• and how about my second quesiton?  maximal likelihood, central limit theorem, law of large numbers e.t.c all fail Jun 25 '21 at 18:37