# Consistency and Asymptotic Normality for MLE of Independent NON-identically distributed normals

I have the following setting: $$x_k \sim N(\mu,\sigma^2 + \hat{\delta}^2_k),k=1,\dots,K,$$

where $\{x_k,k=1,\dots,K\}$ - observed data, $\{\hat{\delta}^2_k\,k=1,\dots,K \}$ are known parameters (just consider them fixed), while $(\mu,\sigma^2)$ are unknown. My primary goal - making inference about $\mu$. Does anyone know of any results for this particular case in terms of consistency and asymptotic behavior of MLE estimator for $\mu$ as $K\rightarrow \infty$?

For the time being I am simply using a $\chi^2$ approximation for the likelihood ratio test, but I don't have solid theoretical argument for that as the data is not identically distributed. It also complicates things that I fail to get a closed-form solution for $\hat{\mu}_{mle}$

I have tried Hoadley's paper on "Asymptotic Properties of MLE for Independent Non-Identically distributed case" http://projecteuclid.org/download/pdf_1/euclid.aoms/1177693066, but verifying their general conditions is not coming to me easy as of yet.

Please let me know if anyone encountered this particular kind of a problem and knows what regularity conditions are needed for nice classic MLE properties. Obviously there should be a certain upper bound on values $\{ \hat{\delta}^2_k,\ k=1,\dots, \}$, but what kind of bound, and even with that bound - how to prove consistency/asymptotic normality.

• The "hat" over your $\delta$s suggests they're estimated from data rather than actually known. Is it the case that you really know the true values rather than estimates of them? Commented May 11, 2017 at 23:34
• You can just consider them fixed/known, yes. The "hat" is more like a notion for them to be distinguished from the usual notation for unknown parameters. Commented May 11, 2017 at 23:57
• ...In the meantime you may want to have a look at stats.stackexchange.com/a/94128/28746 Commented May 20, 2017 at 16:24

You can find much more accessible conditions for consistency and asymptotic normality of MLE in Hayashi's Econometrics, ch. 7.,in the general context of Extremum Estimators and its sub-class, the M-estimators. Hayashi has also references for detailed proofs on the conditions.

The MLE with independent observations belongs to this subclass, because it maximizes a "sample average", an average of a real-valued function of the data and the unknown parameters (note that with independent observations the log-likelihood of the sample is certainly a sum, and we can divide it by the sample size without affecting the solution).

So (in general notation)

$$\hat \theta_{MLE} = \text{argmax}_{\theta} \left\{\frac 1n \sum_{i=1}^n \ell_i(x_i;\theta)\right\}$$

where $\ell_i$ is the log-likelihood of observation $i$.

For consistency, there are two-three alternative sets of conditions. Common to all conditions are:
1) The parameters lie in the interior of the parameter space

2) $\ell_i(x_i;\theta)$ is measurable (if it is continuous, it is measurable)

3) The objective function $\frac 1n \sum_{i=1}^n \ell_i(x_i;\theta)$ converges in probability to some function, say $\ell_0(\theta)$

4) $\ell_0(\theta)$ is uniquely maximized at the true parameter vector (say $\theta_0$)

Then moreover:
1st Alternative : if the parameter space is compact, and convergence is uniform, we obtain consistency.

2nd Alternative : if the parameter space is not compact, then if the log-likelihood is concave and convergence is just pointwise, we again obtain consistency.

I ' ll leave asymptotic normality for the OP to look up and explore.

• I can't find these conditions on the given reference of Hayashi's econometrics chapter 7, it seems to me Hayashi always assumes identically distributed random variables. where could I find this theorem you cite? Commented Apr 1, 2019 at 15:26
• @Monolite Are you sure? See the various theorems in Hayashi pp. 456-465. Commented Apr 1, 2019 at 18:33
• It seems to me that hayashi's notation $Q_n(\theta) := \frac{1}{n} \sum_{t=1}^n \log f(y_t | x_t ; \theta)$ does not assume the $f$ to vary as in your formula $\frac 1n \sum_{i=1}^n \ell_i(x_i;\theta)$ where $\ell_i$ explicitly depends on $i$ . Commented Apr 1, 2019 at 18:46
• @Monolite You are right that Hayashi passes without notice form general data to i.i.d. data. A paper exactly on the case of "independent non-identical" is Bruce Hoadley (1971) "Asymptotic Properties of Maximum Likelihood Estimators for the Independent Not Identically Distributed Case". For non-standard conditions in general, see also stats.stackexchange.com/a/68866/28746 Commented Apr 1, 2019 at 18:59
• @Monolite Yes, it was distilling what one can find in Hayashi, Amemiya, etc Commented Apr 1, 2019 at 19:30