Tossing coin and classical ML estimate I'm reading Bishop's Pattern recognition and came across with the next on the p.23:

Suppose, for instance, that a fair-looking coin is tossed three times
  and lands heads each time. A classical maximum likelihood estimate of
  the probability of landing heads would give 1, implying that all
  future tosses will land heads! By contrast, a Bayesian approach with
  any reasonable prior will lead to a much less extreme conclusion.

Could you explain please, why it is so?
 A: The maximum likelihood estimator for the probality of heads in a binomial distribution is just the proportion of heads, so if in three trials you only get heads, then the maximum likelihood estimate of the probability of heads is 1.  In Bayesian analysis you enhance your estimate with your prior believes. The quote said "fair looking" so a reasonable prior would probably be centered around 0.5, and at least not be centered around 1. This will pull the estimate away from 1. Whether you think that is a good or a bad thing depends on how you feel about using your prior believes in an analysis. This is an subject that has been very "lively" debated.
A: It is true that the MLE in this case is an estimate that the probability of heads is one.  Bear in mind that this is only a point estimate, so it does not necessarily follow that the classical method would assert that all future tosses will land heads with probability one.  More likely the latter prediction would be made using some kind of confidence interval estimate using a pivotal quantity.
It is broadly true that Bayesian analysis incorporates prior information and so it is more likely to give sensible results in these kinds of cases, where we have an a priori belief that the coin should be "close" to fair.  This issue is discussed in O'Neill (2009) where the classical and Bayesian methods are contrasted (using an example of a real-life attempt to determine the fairness of a new coin), and the phenomenon that you are talking about.
