1
$\begingroup$

I'm reading Bishop's Pattern recognition and came across with the next on the p.23:

Suppose, for instance, that a fair-looking coin is tossed three times and lands heads each time. A classical maximum likelihood estimate of the probability of landing heads would give 1, implying that all future tosses will land heads! By contrast, a Bayesian approach with any reasonable prior will lead to a much less extreme conclusion.

Could you explain please, why it is so?

$\endgroup$
4
$\begingroup$

The maximum likelihood estimator for the probality of heads in a binomial distribution is just the proportion of heads, so if in three trials you only get heads, then the maximum likelihood estimate of the probability of heads is 1. In Bayesian analysis you enhance your estimate with your prior believes. The quote said "fair looking" so a reasonable prior would probably be centered around 0.5, and at least not be centered around 1. This will pull the estimate away from 1. Whether you think that is a good or a bad thing depends on how you feel about using your prior believes in an analysis. This is an subject that has been very "lively" debated.

$\endgroup$
  • $\begingroup$ It appears that bayesians always think that probability has distribution. And frequentists think that P=1/1=1, then the next toss gives head, and the next, and the next... Am I right? $\endgroup$ – amplifier Jul 24 '18 at 14:56
  • $\begingroup$ Bayesians think that parameters have a distribution. In this case that parameter happens to be a probability, so now that probability has a distribution. However, it is not generally true Bayesians believe that probabilities have a distribution. Frequentists think that the parameter in the population is fixed, but an estimate from a sample is just one realization from the many possible estimates from all possible samples from that population. The distribution of all possible estimates is the sampling distribution, which is at the core of frequentist inference. $\endgroup$ – Maarten Buis Jul 25 '18 at 7:27
  • $\begingroup$ Notice that the author is trying to emphasize the differences between these, so has chosen an example where that difference is really visible. In real analyses the difference in point estimates is often a lot less dramatic. $\endgroup$ – Maarten Buis Jul 25 '18 at 7:32
1
$\begingroup$

It is true that the MLE in this case is an estimate that the probability of heads is one. Bear in mind that this is only a point estimate, so it does not necessarily follow that the classical method would assert that all future tosses will land heads with probability one. More likely the latter prediction would be made using some kind of confidence interval estimate using a pivotal quantity.

It is broadly true that Bayesian analysis incorporates prior information and so it is more likely to give sensible results in these kinds of cases, where we have an a priori belief that the coin should be "close" to fair. This issue is discussed in O'Neill (2009) where the classical and Bayesian methods are contrasted (using an example of a real-life attempt to determine the fairness of a new coin), and the phenomenon that you are talking about.

$\endgroup$
  • $\begingroup$ thanks you, but I can't open the pdf $\endgroup$ – amplifier Jul 24 '18 at 14:53
  • $\begingroup$ Scholarly articles in academic journals generally require a subscription to the academic publisher that publishes the journal. If you are a university student (or academic) you can usually get access to the article through the university system. Some other organisations that employ researchers also have access to these (e.g., government departments, some big corporations). If you don't fall into any of those categories then you may need to purchase access for an individual article. $\endgroup$ – Reinstate Monica Jul 25 '18 at 0:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.