Posterior distribution and computation of probability of a future event

Question

I am a beginner in statistics, and am self-studying from "Information Theory, Inference, and Learning Algorithms" by David MacKay. I've hit a wall with one of the questions, and was wondering if any of you could be so kind as to point me in the right direction. Admittedly, my title may be wrong or totally misleading as well - if you found it was, please let me know what would have been a better title for this question!

The example which the questions are based on is as follows:

Bill tosses a bent coin N times, obtaining a sequence of heads and tails. We assume that hte coin has a probability fH of coming up heads; we do not know fH. If nH heads have occurred in N tosses, what is the probability distribution of fH? What is the probability that the N+1th outcome will be a head, given nH heads in N tosses.

The question is (this is Exercise 2.8 on p. 30):

Assuming a uniform prior on fH, P(fH) = 1, solve the problem in the example (given above). Sketch the posterior distribution of fH and compute the probability that the N+1th outcome will be a head for

1. N = 3 and nH = 0;
2. N = 3 and nH = 2;
3. N = 10 and nH = 3;
4. N = 300 and nH = 29;

He also states:

You will find the beta integral useful: (Since I'm new I guess I can only give a link to the image - sorry about that): https://i.sstatic.net/7MxQt.png

Now I'm guessing that my difficulty has a lot to do with the fact that I'm new to statistics, and still learning the parlance so to speak. So let me break my question into a few pieces:

1. When he says "Sketch the posterior distribution of fH," what exactly does he mean by that? Does he mean it literally? (Yes...this is how ignorant I am...)
2. Since he doesn't give us fH, is he expecting us to come up with it on our own? Would it be something like (fH + 1)/(N + 2)?
3. First of all, how do I read the beta integral (what exactly is meant by Fa, Fb, pa) and secondly, how is it useful for this problem?

Like I said, I'm a beginner, and trying to grok as much of what he said, but I think at this point I need some help. Thanks in advance!

Answer 1

The notation is horrendous to say the least, e.g. $P(fH)=1$ means $fH$ is uniformly distributed on $(0,1)$?, but if $k$ Heads occurred on $N$ tosses, then the likelihood of this observation is proportional to $$L(k; N, fH) = (fh)^k(1-fH)^{N-k}, ~ 0 < fH < 1.$$ The posterior density function of $fH$ is the likelihood times the prior density function, and since you are told that the $fH$ is uniformly distributed on the unit interval, (that is, the prior density function has value $1$ for $0 \leq fH \leq 1$), the posterior density function of $fH$ is proportional to $L(k; N, fH)$, that is, the posterior density function is $$c\cdot (fh)^k(1-fH)^{N-k}, ~ 0 < fH < 1,$$ and $0$ otherwise. Here $c$ is a constant that ensures that the area under the curve is $1$. In fact, $c$ is given by $$\begin{align*}c &= \frac{1}{\displaystyle\int_0^1 x^k (1-x)^{N-k} \mathrm dx} = \text{Beta}(k+1,N-k+1)\\ &= \frac{\Gamma(N+2)}{\Gamma(k+1)\Gamma(N-k+1)} = \frac{(N+1)!}{k!(N-k)!} \end{align*}$$ So, a sketch of the posterior density is a graph of $c\cdot (fh)^k(1-fH)^{N-k}$ as a function of $fh$, and I suppose what is expected is a curve that has value $0$ at $fH=0$ and $fH=1$ and a single peak at... well, work it out for yourself.

As to what you should estimate as the probability that the $(N+1)$-th toss will result in a Head, some people will defend to the death the value of $fH$ where the posterior density function has a peak, while others will insist that you need to compute the expected value of the random variable whose density is $c\cdot (fh)^k(1-fH)^{N-k}$ (that's where you will need the Beta integral and Beta function) and use that instead. I suspect that your book might be wanting the latter choice.