What happens when you perform MLE on the wrong Probability Distribution Function?

Question

Suppose we have some data sample $x_1, x_2, ... x_n$ that came from some true probability distribution function $f(x; \theta)$.

Based on this sample, i am interested in estimating $\theta$ using Maximum Likelihood Estimation, but I incorrectly assume that it came from some distribution $g(x; \theta)$.

I have the following question about misspecification and robustness:

As my choice of $g(x; \theta)$ deviates further away from $f(x; \theta)$ : How strongly do the properties of the MLE's of $\hat{\theta}$ get affected relative to their true values $\theta$ ?

Here are my guesses:

• Bias: Even under the correct distribution MLE estimates can be biased. Therefore, under the incorrect distribution, the MLE estimates have the possibility of being even more biased?
• Consistency: The MLE estimates for $g(x, \theta)$ will converge in probability (for large samples) to the true values of $g(x, \theta)$ .... but obviously will not converge to the correct values of $f(x, \theta)$
• Asymptotic Normal: For large sample sizes, under the incorrect distribution, the estimates of $g(x, \theta)$ will still be asymptotically normal - but this will be meaningless as they are not cantered around the correct location of $f(x, \theta)$
• Minimum Variance: Again, I think this property will also be respected under the incorrect choice of distribution, but unfortunately this will be meaningless as the distribution choice was incorrect.

Is this the right intuition?

Answer 1

There's a problem before you even get to these questions: how do the $\theta$s match up. That is, you have a family $f_\eta$ and a family $g_\theta$, and some mapping between $\theta$ and $\eta$. This mapping has to be special to get any reasonable answer. I mean, suppose $f$ are $N(\theta,1)$ densities and $g$ are $N(0,\theta)$ densities -- you wouldn't normally call $s^2$ a biased estimate of $\mu$; it's Not Even Wrong.

For this reason it's traditional to talk about an assumed family $g_\theta$ and a single data generating distribution $F$.

There are various things known.

1. Under the same sorts of regularity conditions that you usually use for correctly-specified models, $\hat\theta$ converges to something, which you could call $\theta^*$, or $\theta(F)$. [Huber, or generally, estimating equation theory]
2. Again under the same sorts of conditions, you get $$\sqrt{n}(\hat\theta-\theta^*)\sim N(0,\sigma^2_\theta)$$ [Huber, or generally, estimating equation theory]
3. In general there is no guarantee of anything else really. In particular, there's no guarantee that the map from $\eta$ to the corresponding $\theta$ value, $\theta(F_\eta)$, is one-to-one or smooth or whatever. (eg, my example of $N(\theta,1)$ vs $N(0,\theta)$)
4. However, if $\theta$ and $\eta$ are basically the same sort of parameter so it makes sense, you often will find $\eta\mapsto\theta$ is smooth and invertible and so on -- eg if they're both location parameters
5. It can even be that $\hat\theta\equiv\hat\eta$, for example, the sample average is the MLE of the mean for all exponential family models
6. There's a specially interesting case, which is contiguous misspecification, where you have a sequence $F_n$ getting closer to $G$ as $n$ increases, so the likelihood ration $dF_n/dG_\theta$ is bounded for the 'nearest' $G_\theta$. In that case, we have an exact formula for $\theta^*$ from "LeCam's Third Lemma" [LeCam, but you want to read van der Vaart's Asymptotic Statistics or something, not the original]. There's a bias that's proportional to $dF/dG$ but no change in the variance compared to a correctly-specified model.