# Consistency and Asymptotic Normality for MLE of Independent NON-identically distributed normals

I have the following setting: $$x_k \sim N(\mu,\sigma^2 + \hat{\delta}^2_k),k=1,\dots,K,$$

where $\{x_k,k=1,\dots,K\}$ - observed data, $\{\hat{\delta}^2_k\,k=1,\dots,K \}$ are known parameters (just consider them fixed), while $(\mu,\sigma^2)$ are unknown. My primary goal - making inference about $\mu$. Does anyone know of any results for this particular case in terms of consistency and asymptotic behavior of MLE estimator for $\mu$ as $K\rightarrow \infty$?

For the time being I am simply using a $\chi^2$ approximation for the likelihood ratio test, but I don't have solid theoretical argument for that as the data is not identically distributed. It also complicates things that I fail to get a closed-form solution for $\hat{\mu}_{mle}$

I have tried Hoadley's paper on "Asymptotic Properties of MLE for Independent Non-Identically distributed case" http://projecteuclid.org/download/pdf_1/euclid.aoms/1177693066, but verifying their general conditions is not coming to me easy as of yet.

Please let me know if anyone encountered this particular kind of a problem and knows what regularity conditions are needed for nice classic MLE properties. Obviously there should be a certain upper bound on values $\{ \hat{\delta}^2_k,\ k=1,\dots, \}$, but what kind of bound, and even with that bound - how to prove consistency/asymptotic normality.

• The "hat" over your $\delta$s suggests they're estimated from data rather than actually known. Is it the case that you really know the true values rather than estimates of them? – Glen_b -Reinstate Monica May 11 '17 at 23:34
• You can just consider them fixed/known, yes. The "hat" is more like a notion for them to be distinguished from the usual notation for unknown parameters. – UsDAnDreS May 11 '17 at 23:57
• ...In the meantime you may want to have a look at stats.stackexchange.com/a/94128/28746 – Alecos Papadopoulos May 20 '17 at 16:24

You can find much more accessible conditions for consistency and asymptotic normality of MLE in Hayashi's Econometrics, ch. 7.,in the general context of Extremum Estimators and its sub-class, the M-estimators. Hayashi has also references for detailed proofs on the conditions.

The MLE with independent observations belongs to this subclass, because it maximizes a "sample average", an average of a real-valued function of the data and the unknown parameters (note that with independent observations the log-likelihood of the sample is certainly a sum, and we can divide it by the sample size without affecting the solution).

So (in general notation)

$$\hat \theta_{MLE} = \text{argmax}_{\theta} \left\{\frac 1n \sum_{i=1}^n \ell_i(x_i;\theta)\right\}$$

where $\ell_i$ is the log-likelihood of observation $i$.

For consistency, there are two-three alternative sets of conditions. Common to all conditions are:
1) The parameters lie in the interior of the parameter space

2) $\ell_i(x_i;\theta)$ is measurable (if it is continuous, it is measurable)

3) The objective function $\frac 1n \sum_{i=1}^n \ell_i(x_i;\theta)$ converges in probability to some function, say $\ell_0(\theta)$

4) $\ell_0(\theta)$ is uniquely maximized at the true parameter vector (say $\theta_0$)

Then moreover:
1st Alternative : if the parameter space is compact, and convergence is uniform, we obtain consistency.

2nd Alternative : if the parameter space is not compact, then if the log-likelihood is concave and convergence is just pointwise, we again obtain consistency.

I ' ll leave asymptotic normality for the OP to look up and explore.

• I can't find these conditions on the given reference of Hayashi's econometrics chapter 7, it seems to me Hayashi always assumes identically distributed random variables. where could I find this theorem you cite? – Monolite Apr 1 '19 at 15:26
• @Monolite Are you sure? See the various theorems in Hayashi pp. 456-465. – Alecos Papadopoulos Apr 1 '19 at 18:33
• It seems to me that hayashi's notation $Q_n(\theta) := \frac{1}{n} \sum_{t=1}^n \log f(y_t | x_t ; \theta)$ does not assume the $f$ to vary as in your formula $\frac 1n \sum_{i=1}^n \ell_i(x_i;\theta)$ where $\ell_i$ explicitly depends on $i$ . – Monolite Apr 1 '19 at 18:46
• @Monolite You are right that Hayashi passes without notice form general data to i.i.d. data. A paper exactly on the case of "independent non-identical" is Bruce Hoadley (1971) "Asymptotic Properties of Maximum Likelihood Estimators for the Independent Not Identically Distributed Case". For non-standard conditions in general, see also stats.stackexchange.com/a/68866/28746 – Alecos Papadopoulos Apr 1 '19 at 18:59
• @Monolite Yes, it was distilling what one can find in Hayashi, Amemiya, etc – Alecos Papadopoulos Apr 1 '19 at 19:30