2
$\begingroup$

Suppose we have some data sample $x_1, x_2, ... x_n$ that came from some true probability distribution function $f(x; \theta)$.

Based on this sample, i am interested in estimating $\theta$ using Maximum Likelihood Estimation, but I incorrectly assume that it came from some distribution $g(x; \theta)$.

I have the following question about misspecification and robustness:

As my choice of $g(x; \theta)$ deviates further away from $f(x; \theta)$ : How strongly do the properties of the MLE's of $\hat{\theta}$ get affected relative to their true values $\theta$ ?

Here are my guesses:

  • Bias: Even under the correct distribution MLE estimates can be biased. Therefore, under the incorrect distribution, the MLE estimates have the possibility of being even more biased?
  • Consistency: The MLE estimates for $g(x, \theta)$ will converge in probability (for large samples) to the true values of $g(x, \theta)$ .... but obviously will not converge to the correct values of $f(x, \theta)$
  • Asymptotic Normal: For large sample sizes, under the incorrect distribution, the estimates of $g(x, \theta)$ will still be asymptotically normal - but this will be meaningless as they are not cantered around the correct location of $f(x, \theta)$
  • Minimum Variance: Again, I think this property will also be respected under the incorrect choice of distribution, but unfortunately this will be meaningless as the distribution choice was incorrect.

Is this the right intuition?

$\endgroup$
4
  • $\begingroup$ Glen_b : just re-read and made the edits - thanks! Do you have any opinions about the conclusions I wrote? $\endgroup$ Commented Mar 4 at 13:52
  • 3
  • $\begingroup$ It must depend on what you mean by "deviates" and how you measure the deviation. Your intuition about bias appears flawed: using $g$ might produce solutions that are less biased. Your intuition about consistency is incorrect, because the estimates very well could converge to the correct value of $\theta,$ depending on the relationship between how the two families are parameterized. The asymptotic normality and minimum variance properties also appear dubious: indeed, why should there be any convergence at all when you use the wrong family? $\endgroup$
    – whuber
    Commented Mar 4 at 15:56
  • $\begingroup$ @pnaxso yours is a well-studied problem. Look for "quasi-MLE" on Cross Validated and elsewhere. $\endgroup$
    – Durden
    Commented Mar 4 at 18:33

1 Answer 1

6
$\begingroup$

There's a problem before you even get to these questions: how do the $\theta$s match up. That is, you have a family $f_\eta$ and a family $g_\theta$, and some mapping between $\theta$ and $\eta$. This mapping has to be special to get any reasonable answer. I mean, suppose $f$ are $N(\theta,1)$ densities and $g$ are $N(0,\theta)$ densities -- you wouldn't normally call $s^2$ a biased estimate of $\mu$; it's Not Even Wrong.

For this reason it's traditional to talk about an assumed family $g_\theta$ and a single data generating distribution $F$.

There are various things known.

  1. Under the same sorts of regularity conditions that you usually use for correctly-specified models, $\hat\theta$ converges to something, which you could call $\theta^*$, or $\theta(F)$. [Huber, or generally, estimating equation theory]
  2. Again under the same sorts of conditions, you get $$\sqrt{n}(\hat\theta-\theta^*)\sim N(0,\sigma^2_\theta)$$ [Huber, or generally, estimating equation theory]
  3. In general there is no guarantee of anything else really. In particular, there's no guarantee that the map from $\eta$ to the corresponding $\theta$ value, $\theta(F_\eta)$, is one-to-one or smooth or whatever. (eg, my example of $N(\theta,1)$ vs $N(0,\theta)$)
  4. However, if $\theta$ and $\eta$ are basically the same sort of parameter so it makes sense, you often will find $\eta\mapsto\theta$ is smooth and invertible and so on -- eg if they're both location parameters
  5. It can even be that $\hat\theta\equiv\hat\eta$, for example, the sample average is the MLE of the mean for all exponential family models
  6. There's a specially interesting case, which is contiguous misspecification, where you have a sequence $F_n$ getting closer to $G$ as $n$ increases, so the likelihood ration $dF_n/dG_\theta$ is bounded for the 'nearest' $G_\theta$. In that case, we have an exact formula for $\theta^*$ from "LeCam's Third Lemma" [LeCam, but you want to read van der Vaart's Asymptotic Statistics or something, not the original]. There's a bias that's proportional to $dF/dG$ but no change in the variance compared to a correctly-specified model.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.