# Convergence of MLE for non-IID data

Consider calculating optimal model parameter $$\theta$$ using MLE for the following 2 cases:

1. Data generating process is independent but non identical:

$$L(y;\theta) = \prod_{i=1}^{n} f_{i}(y_i;\theta)$$.

1. Data generating process has sequential dependence and could be non-stationary:

$$L(y;\theta) = \prod_{i=2}^{n} f_{i}(y_i|y_{i-};\theta) f_{1}(y_1)$$.

The subscript $$i$$ is to indicate that the distribution of each term $$f$$ could be distinct. If the model is well specified, would MLE converge for 1. and 2.? Are there any proofs for convergence (or lack of) for 1. and 2. ?

• Model (2) is far too general to permit any conclusions to be made. Do you have a specific model in mind?
– whuber
Commented Jul 2 at 21:39
• @whuber specifically thinking of transition densities of stochastic differential equations ex. GBM or OU processes. Commented Jul 3 at 10:01

It will depend, but there are some things that can usefully be said, especially if this is a smooth parametric model (as seems to be implied)

If you have a law of large numbers and central limit theorem for $$\log f_i$$ and the usual smoothness assumptions, then the classical proofs of consistency and asymptotic normality transfer over if you just swap out the existing LLN and CLT for a new one.

There are lots of non-independent CLTs out there, eg

• $$\log f_i-E\,[\log f_i|\text{history}]$$ is a martingale or a local martingale
• mixing conditions on $$y$$ as a stochastic process
• mixing conditions on $$y$$ as a random field
• 'sparse' correlation conditions (Stein's method for the CLT, "graph-structured dependence" is a useful keyword)

Sadly, there is a tendency for the more complicated CLTs to be stated for identically distributed $$y_i$$ (eg, stationary sequences) because probabilists want to separate the interesting complications from dependence and the boring complications from differing distributions.

However, it is typically the case that (as for independent sequences) tail conditions plus conditions that all the terms are very roughly the same size will get you from identical distributions to non-identical distributions.

In the setting of time series there are some results under "long-range dependence" that don't just come from the classical proofs.

• I had a question. Would IID error ($y_{data} - y_{model}$) distribution for both cases in the question be a sufficient condition for convergence? Commented Jul 28 at 20:01
• No, it's not sufficient: you need at least one finite moment for $\log f_i$ to get convergence of the loglikelihood, and at least a little bit of uniformity in $\theta$ for the convergence to translate into consistency of $\hat\theta$ Commented Jul 29 at 2:05
• Makes sense. The $\log(f_i)$ in the comment above refers to both cases in the question right? Commented Jul 29 at 8:36
• yes, it will be the marginal or conditional moments as appropriate Commented Jul 29 at 21:56
• Thank you. Can one consider regression model to fall under the category of case 1 (INID case) as we usually have a model with fixed conditional variance and some linear/non-linear function for the conditional mean ? $L(D;\theta) = \sum_{i=1}^{N} \log(p_{Y|X}(x_i, y_i)) = \sum_{i=1}^{N} \log(N(f(x), \sigma^{2}))$ where $N$ represents a gaussian distribution and $f$ is some ML/NN model to estimate the mean. As the mean is varying, the distribution of each term in the log likelihood is not fixed. Commented Jul 30 at 7:32