Statistical learning when observations are not iid

Question

As far as I am concerned, statistical/machine learning algorithms always suppose that data are independent and identically distributed ($iid$).

My question is: what can we do when this assumption is clearly unsatisfied? For instance, suppose that we have a data set whith repeated measurements on the same observations , so that both the cross-section and the time dimensions are important (what econometricians call a panel data set, or statisticians refer to as longitudinal data, which is distinct from a time series).

An example could be the following. In 2002, we collect the prices (henceforth $Y$) of 1000 houses in New York, together with a set of covariates (henceforth $X$). In 2005, we collect the same variables on the same houses. Similar happens in 2009 and 2012. Say I want to understand the relationship between $X$ and $Y$. Were the data $iid$, I could easily fit a random forest (or any other supervised algorithm, for what matters), thus estimating the conditional expectation of $Y$ given $X$. However, there is clearly some auto-correlation in my data. How can I handle this?

Answer 1

There is nothing in the theory of statistical learning or machine learning that requires samples to be i.i.d.

When samples are i.i.d, you can write the joint probability of the samples given some model as a product, namely $P(\{x\}) = \Pi_{i} P_i(x_i)$ which makes the log-likelihood a sum of the individual log-likelihoods. This simplifies the calculation, but is by no means a requirement.

In your case, you can for example model the distribution of a pair $x_i,y_i$ with some bi-variate distribution, say $z_i=(x_i,y_i)^T$ , $z_i \sim \mathcal{N}(\mu,\Sigma)$ , and then estimate the parameter $\Sigma$ from the likelihood $P(\{z\}) = \Pi_{i} P(z_i | \mu, \Sigma)$.

It is true that many out-of-the-box algorithm implementations implicitly assume independence between samples, so you are correct in identifying that you will have a problem applying them to you data as is. You will either have to modify the algorithm or find ones that are better suited for your case.

Answer 2

Markov processes are not only very general ways to analyze longitudinal data with statistical models, they also lend themselves to machine learning. They work because by modeling transition probabilities conditional on previous states the records are conditionally independent and may be treated as coming from different independent subjects. One can use discrete or continuous time processes, discrete being simpler. The main work comes from post-estimation processing to convert transition probabilities into unconditional (on previous state) state occupancy probabilities AKA current status probabilities. See this and other documents in this.

Answer 3

There are a few good answers here already but I thought it worth noting that the answer to this question can change drastically depending on how the iid assumption is violated. For example, if a univariate dataset is not iid, but is stationary, then many very simple estimation procedures, such as the sample mean, still converge to the appropriate limit.

However, if the iid assumption is violated because the data is non-stationary, then life is much more difficult. Note that the very common Machine Learning tradition of splitting the dataset into a training, test, and sometimes validation set, is invalid in the presence of non-stationarity. If this is the difficulty you face then your best bet is usually to try and find a transformation of the data that is close to stationary (or ergodicity) and work with that instead.