29
$\begingroup$

I have a confusion on biased maximum likelihood (ML) estimators. The mathematics of the whole concept is pretty clear to me but I cannot figure out the intuitive reasoning behind it.

Given a certain dataset which has samples from a distribution, which itself is a function of a parameter that we want to estimate, the ML estimator results in the value for the parameter which is most likely to produce the dataset.

I cannot intuitively understand a biased ML estimator in the sense that: how can the most likely value for the parameter predict the real value of the parameter with a bias towards a wrong value?

$\endgroup$
2

4 Answers 4

17
$\begingroup$

the ML estimator results in the value for the parameter which is most likely to occur in the dataset.

Given the assumptions, the ML estimator is the value of the parameter that has the best chance of producing the data set.

I cannot intuitively understand a biased ML estimator in the sense that "How can the most likely value for the parameter predict the real value of the parameter with a bias towards a wrong value?"

Bias is about expectations of sampling distributions. "Most likely to produce the data" isn't about expectations of sampling distributions. Why would they be expected to go together?

What is the basis on which it is surprising they don't necessarily correspond?

I'd suggest you consider some simple cases of MLE and ponder how the difference arises in those particular cases.

As an example, consider observations on a uniform on $(0,\theta)$. The largest observation is (necessarily) no bigger than the parameter, so the parameter can only take values at least as large as the largest observation.

When you consider the likelihood for $\theta$, it is (obviously) larger the closer $\theta$ is to the largest observation. So it's maximized at the largest observation; that's clearly the estimate for $\theta$ that maximizes the chance of obtaining the sample you got:

enter image description here

But on the other hand it must be biased, since the largest observation is obviously (with probability 1) smaller than the true value of $\theta$; any other estimate of $\theta$ not already ruled out by the sample itself must be larger than it, and must (quite plainly in this case) be less likely to produce the sample.

The expectation of the largest observation from a $U(0,\theta)$ is $\frac{n}{n+1}\theta$, so the usual way to unbias it is to take as the estimator of $\theta$: $\hat\theta=\frac{n+1}{n}X_{(n)}$, where $X_{(n)}$ is the largest observation.

This lies to the right of the MLE, and so has lower likelihood.

$\endgroup$
4
  • $\begingroup$ thanks for your answer. About the first part, I expressed myself incorrectly. I basically meant what you said. Based on your answer to the second part, can I conclude that given another set of data drawn from the same distribution, will the ML estimator result in a different bias? Since you say that the ML estimator is the one which "most likely" produce the data. If we change the data some other estimator might most likely produce it. Is that correct? $\endgroup$
    – ssah
    Mar 4, 2014 at 23:08
  • $\begingroup$ The estimator won't change if the form of the population distribution doesn't change. Some other estimate will be produced with a different sample and the amount by which it is biased will generally be different -- bias is usually related to sample size, even if the population is the same. ...(ctd) $\endgroup$
    – Glen_b
    Mar 4, 2014 at 23:24
  • $\begingroup$ (ctd)... $\quad$ Note that I've made some edits above which might help. In the context of my above example, with a different sample (this time of size $m$ rather than $n$, say) - the form of the ML estimator would still be 'the largest observation in the sample', but the estimate would be different (even with the same $\theta$), and the bias would also typically be different (because of the sample size effect). $\endgroup$
    – Glen_b
    Mar 4, 2014 at 23:25
  • $\begingroup$ Good use of the canonical example for seeing the difference between unbiased and ML estimators. $\endgroup$
    – jwg
    Mar 5, 2014 at 11:03
8
$\begingroup$

$\beta^{MLE}$ is not the most probable value of $\beta$. The most probable value is $\beta $ itself. $\beta^{MLE}$ maximizes the probability of drawing the sample that we actually got.

MLE is only asymptotically unbiased, and often you can adjust the estimator to behave better in finite samples. For example, the MLE of the variance of a random variable is one example, where multiplying by $\frac{N}{N-1}$ transforms it.

$\endgroup$
4
  • $\begingroup$ Sorry for the mistake in the first part. I edited and fixed it. But about what you said about the MLE, why would it be biased at the first place in the non asymptotic case? $\endgroup$
    – ssah
    Mar 4, 2014 at 23:14
  • 2
    $\begingroup$ "Better" depends on what you look at; Bessel's correction makes it unbiased, but unbiasedness is not of itself automatically "better" (the MSE is worse, for example; why should I prefer unbiasedness to smaller MSE?). Unbiasedness might be argued to be better, ceteris paribus, but unfortunately the ceteris won't be paribus. $\endgroup$
    – Glen_b
    Mar 4, 2014 at 23:29
  • $\begingroup$ My understanding was that the unbiased estimator can be shown to be best unbiased through the relationship between the MLE and the Cramer-Rao lower bound. $\endgroup$
    – dimitriy
    Mar 5, 2014 at 0:36
  • $\begingroup$ @ssah I have been told that it is because we are using the sample mean instead of the true mean in the formula. To be honest, I've never really found this explanation particularly intuitive, because if the MLE estimator of the mean is unbiased, why should this go wrong? I usually put my doubts to rest with a simulation. $\endgroup$
    – dimitriy
    Mar 5, 2014 at 0:52
7
$\begingroup$

Here's my intuition.

Bias is a measure of accuracy, but there's also a notion of precision.

enter image description here

In an ideal world, we'd get the estimate, which is both precise and accurate, i.e. always hits the bull's eye. Unfortunately, in our imperfect world, we have to balance accuracy and precision. Sometimes we may feel that we could give a bit of accuracy to gain more precision: we trade-off all the time. Hence, the fact that an estimator is biased doesn't mean that it's bad: it could be that it's more precise.

$\endgroup$
3
$\begingroup$

The ordinary-language and technical meanings of biased are different. The answer by @Glen_b gives a good description of why maximum likelihood estimators can easily be biased in the technical sense.

It is possible for the maximum likelihood estimator to be biased in something like the ordinary-language sense, but it's not usual. Something has to go wrong.

The standard examples of an inconsistent MLE involve paired data.

Suppose $X_{ij}\sim N(\mu_i,\sigma^2)$, for $j=0,1$, and $i=1,2,3,\dots,n$. The MLE of $\mu_i$ is $(X_{i0}+X_{i1})/2$. The MLE of $\hat\sigma^2$ is $$\hat\sigma^2= \frac{1}{2n}\sum_{i=1}^n\sum_{j=0}^1 (X_{ij}-\hat\mu_i)^2$$

As you get more data, $\hat\sigma^2$ converges not to $\sigma^2$ but to $\sigma^2/2$.

With binary matched-pair data the generating model is $$\mathrm{logit}\,P[Y_{ij}=1]=\alpha_i+\beta X_{ij}$$ The MLE $\hat\beta$ converges to $2\beta$ rather than to $\beta$.

In both cases, the problem is having the number of parameters grow with $n$, and the solution is a conditional likelihood that removes the $n$ intercept parameters before estimating the parameter you're interested in.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.