2
$\begingroup$

Lehman's Element's of Statistical Learning Theory gives in Theorem 7.5.2 a central limit theorem for multiparamter maximum likelihood estimators. (Many other sources provide similar theorems.) The theorem states that under certain technical conditions,

$$ \sqrt{n}(\theta^* - \theta_n) \rightarrow_L \mathcal{N}(0,I(\theta^*)^{-1}) $$

where $\theta^*$ is the true parameter vector, $\theta_n$ is the parameter vector estimated from $n$ samples, and $I(\theta^*)$ is the Fischer information matrix.

The technical conditions of these theorems never explicitly state that the model whose parameters we are trying to learn is a good model for the data in any sense. But the examples of using the theorem always assume this. For example, immediately after stating the theorem above, Lehman uses this theorem to prove that the mean and variance estimates of a normal distribution are themselves normally distributed. But the example assumes that the data points are normally distributed.

What if the data were actually exponentially distributed, but I make a horrible modelling assumption that the data is normally distributed? Does the CLT for MLE still hold? In general, is there a way to characterize the distribution of the parameters that depends on how poor of a modelling assumption we've made?

Improve this question
$\endgroup$
3
  • $\begingroup$ In a sense yes, if the right conditions are met. In your example the MLE will converge to a single true value and will be asymptotically normally distributed around it, but the problem is what that value represents (in the particular example $\theta^*=1/\lambda$, where $\lambda$ is the exponential rate). Additionally, for some things the convergence will be much slower (e.g. the odds ratio from a randomized controlled trial needs a much larger sample size to be reasonably normally distributed than the log-odds ratio). $\endgroup$
    – Björn
    Commented Feb 23, 2016 at 5:33
  • 1
    $\begingroup$ What you may be after is quasi maximum likelihood estimation. See this questions and especially the three references therein (both the Wiki, the textbook and White's paper are good references). $\endgroup$
    – Richard Hardy
    Commented Feb 23, 2016 at 7:18
  • $\begingroup$ @RichardHardy White's paper in the link you mention is exactly what I was looking for! Thanks! $\endgroup$
    – Mike Izbicki
    Commented Feb 23, 2016 at 22:31

2 Answers 2

Reset to default
1
$\begingroup$

It turns out that the sample mean and sample variance, which are the sufficient statistics for the two parameter normal, $\theta = [\mu, \sigma^2]$, are also unbiased consistent estimators of the mean and variance of any other distribution by the WLLN (provided those moments exist) and have a limiting normal distribution that is the same as that of the ML theory by the central limit theorem (when regularity conditions are met). So for your example, normal probability model imposed on exponentially distributed data, you are okay. The normal estimates are the method of moments estimators.

I don't think as much can be said in general. If the underlying data were shifted/scaled Cauchy, $\mathcal{T}_1(\delta)$, the mean is an inconsistent estimator of the shift parameter. If the underlying data were $\mathcal{Beta}(\alpha, \beta)$ and we imposed a 1 parameter model: $\mathcal{Beta}(\alpha^*, 1)$ the $\alpha^*$ would have some estimand and a limiting distribution around it that is not necessarily $\alpha$ but a function of $\beta$ and $\alpha$; this is a justification for quasilikelihood. If the underlying data were double exponential. The normal distribution would be unbiased in estimating the mean, but it would project a variance that is too small because the median is inefficient relative to the mean.

I think these three "counterexamples" show why quasilikelihood has a basis: most sufficient statistics are also method of moments statistics and have CLTs for other reasons, and estimate useful things. However, in ML theory, when you know the probability model, you get more efficient and powerful estimators nearly every time.

Improve this answer
$\endgroup$
1
$\begingroup$

I Googled it, and was pleasantly surprised to stumble upon my own talk from more than a decade ago :). I know that one-line link responses are awful, but this topic is so dense that it would be impossible to reproduce all the arguments, short of pasting the complete $\LaTeX$ code here.

http://staskolenikov.net/talks/Stas-misspecified-handout.pdf

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.