Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a confusion on biased Maximum Likelihood estimators. The mathematics of the whole concept is pretty clear to me but I cannot figure out the intuitive reasoning behind it. Given a certain dataset which has samples from a distribution which itself is a function of a parameter that we want to estimate, the ML estimator results in the value for the parameter which is most likely to produce the dataset. I cannot intuitively understand a biased ML estimator in the sense that "How can the most likely value for the parameter predict the real value of the parameter with a bias towards a wrong value?"

share|improve this question

the ML estimator results in the value for the parameter which is most likely to occur in the dataset.

Given the assumptions, the ML estimator is the value of the parameter that has the best chance of producing the data set.

I cannot intuitively understand a biased ML estimator in the sense that "How can the most likely value for the parameter predict the real value of the parameter with a bias towards a wrong value?"

Bias is about expectations of sampling distributions. "Most likely to produce the data" isn't about expectations of sampling distributions. Why would they be expected to go together?

What is the basis on which it is surprising they don't necessarily correspond?

I'd suggest you consider some simple cases of MLE and ponder how the difference arises in those particular cases.

As an example, consider observations on a uniform on $(0,\theta)$. The largest observation is (necessarily) no bigger than the parameter, so the parameter can only take values at least as large as the largest observation.

When you consider the likelihood for $\theta$, it is (obviously) larger the closer $\theta$ is to the largest observation. So it's maximized at the largest observation; that's clearly the estimate for $\theta$ that maximizes the chance of obtaining the sample you got:

enter image description here

But on the other hand it must be biased, since the largest observation is obviously (with probability 1) smaller than the true value of $\theta$; any other estimate of $\theta$ not already ruled out by the sample itself must be larger than it, and must (quite plainly in this case) be less likely to produce the sample.

The expectation of the largest observation from a $U(0,\theta)$ is $\frac{n}{n+1}$, so the usual way to unbias it is to take as the estimator of $\theta$: $\hat\theta=\frac{n+1}{n}X_{(n)}$, where $X_{(n)}$ is the largest observation.

This lies to the right of the MLE, and so has lower likelihood.

share|improve this answer
thanks for your answer. About the first part, I expressed myself incorrectly. I basically meant what you said. Based on your answer to the second part, can I conclude that given another set of data drawn from the same distribution, will the ML estimator result in a different bias? Since you say that the ML estimator is the one which "most likely" produce the data. If we change the data some other estimator might most likely produce it. Is that correct? – ssah Mar 4 '14 at 23:08
The estimator won't change if the form of the population distribution doesn't change. Some other estimate will be produced with a different sample and the amount by which it is biased will generally be different -- bias is usually related to sample size, even if the population is the same. ...(ctd) – Glen_b Mar 4 '14 at 23:24
(ctd)... $\quad$ Note that I've made some edits above which might help. In the context of my above example, with a different sample (this time of size $m$ rather than $n$, say) - the form of the ML estimator would still be 'the largest observation in the sample', but the estimate would be different (even with the same $\theta$), and the bias would also typically be different (because of the sample size effect). – Glen_b Mar 4 '14 at 23:25
Good use of the canonical example for seeing the difference between unbiased and ML estimators. – jwg Mar 5 '14 at 11:03

$\beta^{MLE}$ is not the most probable value of $\beta$. The most probable value is $\beta $ itself. $\beta^{MLE}$ maximizes the probability of drawing the sample that we actually got.

MLE is only asymptotically unbiased, and often you can adjust the estimator to behave better in finite samples. For example, the MLE of the variance of a random variable is one example, where multiplying by $\frac{N}{N-1}$ transforms it.

share|improve this answer
Sorry for the mistake in the first part. I edited and fixed it. But about what you said about the MLE, why would it be biased at the first place in the non asymptotic case? – ssah Mar 4 '14 at 23:14
"Better" depends on what you look at; Bessel's correction makes it unbiased, but unbiasedness is not of itself automatically "better" (the MSE is worse, for example; why should I prefer unbiasedness to smaller MSE?). Unbiasedness might be argued to be better, ceteris paribus, but unfortunately the ceteris won't be paribus. – Glen_b Mar 4 '14 at 23:29
My understanding was that the unbiased estimator can be shown to be best unbiased through the relationship between the MLE and the Cramer-Rao lower bound. – Dimitriy V. Masterov Mar 5 '14 at 0:36
@ssah I have been told that it is because we are using the sample mean instead of the true mean in the formula. To be honest, I've never really found this explanation particularly intuitive, because if the MLE estimator of the mean is unbiased, why should this go wrong? I usually put my doubts to rest with a simulation. – Dimitriy V. Masterov Mar 5 '14 at 0:52

here's my intuition. bias is a measure of accuracy, but there's also a notion of precision. in an ideal world we'd get the estimate, which is both precise and accurate, i.e. always hits the bull's eye. unfortunately, in our imperfect world we have to balance accuracy and precision. sometimes we may feel that we could give a bit of accuracy to gain more precision. we trade-off all the time. hence, the fact that an estimator is biased doesn't mean that it's bad. it could be that it's more precise.

enter image description here

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.