Bayes theorem with multiple draws

Question

Setting

I have a question on the "Cookie Problem Revisited" exercise from Allen Downey's Think Bayes 2e. The Bayes theorem is defined as:

$$ P(H \mid E) = \frac{P(H) \ P(E \mid H)}{P(E)} $$

where $E$ is the evidence and H is the hypothesis.

Let's consider the following examples: two bowls contain different cookies.

• Bowl 1 contains 30 vanilla cookies and 10 chocolate cookies.
• Bowl 2 contains 20 vanilla cookies and 20 chocolate cookies.

We randomly chose a bowl and draw a vanilla cookie. What is the probability that it came from Bowl 1?

$P(H=\text{Bowl 1}) = 0.5$ (we randomly select the bowl)

$P(E=\text{vanilla} \mid H=\text{Bowl 1}) = 30/(30+10) = 3/4 $

$P(E=\text{vanilla}) = (30 + 20)/(30 + 20 + 10 + 20) = 5/8$ (vanilla cookies over total cookies)

Then, the posterior $P(H=\text{Bowl 1} \mid E=\text{vanilla}) = 0.6 $.

Question

Now we place the vanilla cookie back into the bowl and draw a second cookie. We get vanilla again. We can use the posterior we have just calculated as the new prior. The posterior is now: $P(E=\text{vanilla} \mid H=\text{Bowl 1}) \times P(H=\text{previous posterior})$ as calculated earlier. However, the probability of the evidence must be different from what we computed earlier (vanilla cookies over total cookies), because otherwise the posterior for Bowl 1 and Bowl 2 would not sum up to 1 (I haven't expressed the posterior for Bowl 2 in this text to keep it simple).

Why isn't the probability of the evidence equal to the ratio of vanilla cookies over total cookies when we draw multiple times? And how can we calculate it in this case?

Answer 1

Suppose that you have information from two draws let's say $E_{1}=vanilla$ and $E_{2}=vanilla$, and you want to calculate the probability of coming from the first bowl, then you will have

$$p(H=B_{1}|E_{1}=vanilla,E_{2}=vanilla)=\frac{p(E_{2}=vanilla|H=B_{1},E_{1}=vanilla)p(H=B_{1}|E_{1}=vanilla)}{p(E_{2}=vanilla|E_{1}=vanilla)}$$

$E_{2}$ is independent of $E_{1}$ conditional on $H=B_{1}$, hence you have

$$p(H=B_{1}|E_{1}=vanilla,E_{2}=vanilla)=\frac{p(E_{2}=vanilla|H=B_{1})p(H=B_{1}|E_{1}=vanilla)}{p(E_{2}=vanilla|E_{1}=vanilla)}$$

where you can calculate the $p(E_{2}=vanilla|E_{1}=vanilla)$ with the law of total probability which eventually will be equal to $5/8$ because you have replacement

Answer 2

Why isn't the probability of the evidence equal to the ratio of vanilla cookies over total cookies when we draw multiple times? And how can we calculate it in this case?

The probability of the evidence depends on the prior

$$E(\text{vanilla}) = P(\text{bowl 1}) \times P(\text{vanilla}\mid \text{bowl 1}) + P(\text{bowl 2}) \times P(\text{vanilla}\mid \text{bowl 2})$$

In your method of computation, the prior changes over time. After the first draw of a vanilla cookie, the vanilla in a second draw becomes more likely.

An extreme example might make this more clear, imagine you had a hundred draws and 78 cookies were vanilla and 22 cookies were chocolate. Then wouldn't you likely have bowl 1 and shouldn't the probability of a vanilla draw be closer to 3/4 rather than 5/8?

This probability for vanilla relates to the posterior predictive probability with the posterior from the previous draw.

---

You can also compute the probabilities for the four different ways of making two draws

$$\begin{array}{} P(vv) &=& 0.5 \frac{1}{4} + 0.5 \frac{9}{16} = \frac{13}{32}\\ P(cv) &=& 0.5 \frac{1}{4} + 0.5 \frac{3}{16} = \frac{7}{32}\\ P(vc) &=& 0.5 \frac{1}{4} + 0.5 \frac{3}{16} = \frac{7}{32}\\ P(cc) &=& 0.5 \frac{1}{4} + 0.5 \frac{1}{16} = \frac{5}{32}\\ \end{array}$$

and the odds for '1st vanilla 2nd vanilla' versus '1st vanilla 2nd chocolate' are 13:7 instead of 5:3.