# What happens when you perform MLE on the wrong Probability Distribution Function?

Suppose we have some data sample $$x_1, x_2, ... x_n$$ that came from some true probability distribution function $$f(x; \theta)$$.

Based on this sample, i am interested in estimating $$\theta$$ using Maximum Likelihood Estimation, but I incorrectly assume that it came from some distribution $$g(x; \theta)$$.

I have the following question about misspecification and robustness:

As my choice of $$g(x; \theta)$$ deviates further away from $$f(x; \theta)$$ : How strongly do the properties of the MLE's of $$\hat{\theta}$$ get affected relative to their true values $$\theta$$ ?

Here are my guesses:

• Bias: Even under the correct distribution MLE estimates can be biased. Therefore, under the incorrect distribution, the MLE estimates have the possibility of being even more biased?
• Consistency: The MLE estimates for $$g(x, \theta)$$ will converge in probability (for large samples) to the true values of $$g(x, \theta)$$ .... but obviously will not converge to the correct values of $$f(x, \theta)$$
• Asymptotic Normal: For large sample sizes, under the incorrect distribution, the estimates of $$g(x, \theta)$$ will still be asymptotically normal - but this will be meaningless as they are not cantered around the correct location of $$f(x, \theta)$$
• Minimum Variance: Again, I think this property will also be respected under the incorrect choice of distribution, but unfortunately this will be meaningless as the distribution choice was incorrect.

Is this the right intuition?

• Glen_b : just re-read and made the edits - thanks! Do you have any opinions about the conclusions I wrote? Commented Mar 4 at 13:52
• Similar earlier questions: Statistical Inference Under Misspecification, Maximum likelihood estimation when the model is misspecified (and the true data generating process is a mixture model) and also search this site Commented Mar 4 at 14:00
• It must depend on what you mean by "deviates" and how you measure the deviation. Your intuition about bias appears flawed: using $g$ might produce solutions that are less biased. Your intuition about consistency is incorrect, because the estimates very well could converge to the correct value of $\theta,$ depending on the relationship between how the two families are parameterized. The asymptotic normality and minimum variance properties also appear dubious: indeed, why should there be any convergence at all when you use the wrong family?
– whuber
Commented Mar 4 at 15:56
• @pnaxso yours is a well-studied problem. Look for "quasi-MLE" on Cross Validated and elsewhere. Commented Mar 4 at 18:33

## 1 Answer

There's a problem before you even get to these questions: how do the $$\theta$$s match up. That is, you have a family $$f_\eta$$ and a family $$g_\theta$$, and some mapping between $$\theta$$ and $$\eta$$. This mapping has to be special to get any reasonable answer. I mean, suppose $$f$$ are $$N(\theta,1)$$ densities and $$g$$ are $$N(0,\theta)$$ densities -- you wouldn't normally call $$s^2$$ a biased estimate of $$\mu$$; it's Not Even Wrong.

For this reason it's traditional to talk about an assumed family $$g_\theta$$ and a single data generating distribution $$F$$.

There are various things known.

1. Under the same sorts of regularity conditions that you usually use for correctly-specified models, $$\hat\theta$$ converges to something, which you could call $$\theta^*$$, or $$\theta(F)$$. [Huber, or generally, estimating equation theory]
2. Again under the same sorts of conditions, you get $$\sqrt{n}(\hat\theta-\theta^*)\sim N(0,\sigma^2_\theta)$$ [Huber, or generally, estimating equation theory]
3. In general there is no guarantee of anything else really. In particular, there's no guarantee that the map from $$\eta$$ to the corresponding $$\theta$$ value, $$\theta(F_\eta)$$, is one-to-one or smooth or whatever. (eg, my example of $$N(\theta,1)$$ vs $$N(0,\theta)$$)
4. However, if $$\theta$$ and $$\eta$$ are basically the same sort of parameter so it makes sense, you often will find $$\eta\mapsto\theta$$ is smooth and invertible and so on -- eg if they're both location parameters
5. It can even be that $$\hat\theta\equiv\hat\eta$$, for example, the sample average is the MLE of the mean for all exponential family models
6. There's a specially interesting case, which is contiguous misspecification, where you have a sequence $$F_n$$ getting closer to $$G$$ as $$n$$ increases, so the likelihood ration $$dF_n/dG_\theta$$ is bounded for the 'nearest' $$G_\theta$$. In that case, we have an exact formula for $$\theta^*$$ from "LeCam's Third Lemma" [LeCam, but you want to read van der Vaart's Asymptotic Statistics or something, not the original]. There's a bias that's proportional to $$dF/dG$$ but no change in the variance compared to a correctly-specified model.