Convergence of MLE for non-IID data

Question

Consider calculating optimal model parameter $\theta$ using MLE for the following 2 cases:

1. Data generating process is independent but non identical:

$L(y;\theta) = \prod_{i=1}^{n} f_{i}(y_i;\theta)$.

2. Data generating process has sequential dependence and could be non-stationary:

$L(y;\theta) = \prod_{i=2}^{n} f_{i}(y_i|y_{i-};\theta) f_{1}(y_1)$.

The subscript $i$ is to indicate that the distribution of each term $f$ could be distinct. If the model is well specified, would MLE converge for 1. and 2.? Are there any proofs for convergence (or lack of) for 1. and 2. ?

Answer 1

It will depend, but there are some things that can usefully be said, especially if this is a smooth parametric model (as seems to be implied)

If you have a law of large numbers and central limit theorem for $\log f_i$ and the usual smoothness assumptions, then the classical proofs of consistency and asymptotic normality transfer over if you just swap out the existing LLN and CLT for a new one.

There are lots of non-independent CLTs out there, eg

• $\log f_i-E\,[\log f_i|\text{history}]$ is a martingale or a local martingale
• mixing conditions on $y$ as a stochastic process
• mixing conditions on $y$ as a random field
• 'sparse' correlation conditions (Stein's method for the CLT, "graph-structured dependence" is a useful keyword)

Sadly, there is a tendency for the more complicated CLTs to be stated for identically distributed $y_i$ (eg, stationary sequences) because probabilists want to separate the interesting complications from dependence and the boring complications from differing distributions.

However, it is typically the case that (as for independent sequences) tail conditions plus conditions that all the terms are very roughly the same size will get you from identical distributions to non-identical distributions.

In the setting of time series there are some results under "long-range dependence" that don't just come from the classical proofs.