Bayesian updating with discrete priors + possibly unknown classes

Question

I'm following along with some lecture notes on Bayesian updating with discrete priors. They give an example problem to illustrate some of these concepts, which I briefly restate here:

Someone tells you they have three types of coins (A, B, and C) each with different probabilities of HEADS:

• Coin Type A has p = 0.5,
• Coin Type B has p = 0.6, and
• Coin Type C has p = 0.9.

Determine the posterior probability for each coin type, given that the outcome of a single toss is HEADS.

Actually, the authors give you priors as well:

The coin being tossed is drawn at random from a desk drawer containing two A's, two B's, and one C.

The solution is quite simple: Multiply the likelihoods (in this case, {0.5, 0.6, 0.9}) by the priors {0.4, 0.4, 0.2}, then normalize. In fact, pretty much every set of notes I've read on the subject contains the following expression:

$$posterior \,\propto\, likelihood \, \times \, prior $$

To test out some of these ideas, I wrote a quick simulation to see how the posteriors behave after observing $N$ outcomes of coin tosses. In order to make this problem more like a realistic scenario (and less trivial), I'd now like to consider a modified problem: There is now a fourth category of coin, D, whose probability of heads P(HEADS|D) is unknown.

The formula quoted above now seems to break, so my question is, How does a Bayesian proceed on the modified problem?

One approach I've considered is parameter estimation: After each new coin toss, use all the observed data to estimate the population parameter of a binomial distribution. Then successively test each of four hypotheses:

1. $\mathcal{H}_1 =$ Coin is type A
2. $\mathcal{H}_2 =$ Coin is type B
3. $\mathcal{H}_3 =$ Coin is type C
4. $\mathcal{H}_4 =$ Coin is neither type A, B, nor C

But this seems like such a radically different approach, and awfully frequentist. I'm looking for a method that is a simple modification of the method above, or, failing that, a different method that doesn't altogether abandon Bayesian concepts like priors, posteriors, and likelihoods.

Thanks for reading

Answer 1

Ok, as it turns out I answered my own question. I want to think of this problem as finding the posterior probability

$$p(H_i|x) = \frac{ p(x|H_i) p(H_i)}{p(x)} $$

but I've found a situation where I'm not sure what to use for the likelihood function in the case where $H_n = \text{unknown coin}$. As explained below, my confusion stemmed from the fact that there are two meanings of priors here.

Resolution

To make this problem very simple, I have reduced it to only two hypotheses:

• $H_0$ = the coin is fair, i.e., $p^H = 0.5$
• $H_1$ = the coin is not fair, or $p^H \neq \,0.5$

And to further simplify, I'll assume that either the coins are mixed in equal proportions in the desk drawer, or else that I was never told; in either case, the prior probabilities of each hypothesis are equal, $p(H_0) = p(H_1) = 0.5$ to begin with. After observing any particular coin toss, the likelihood function $p(x|H_0)$ is the same regardless of the outcome: $$p(x=\text{heads}|H_0) = p(x=\text{tails}|H_0) = 0.5$$

We also do not really care about $p(x)$ since it will be a normalizing constant. The only ingredient left is $p(x|H_1)$. The trick is to consider the likelihood function as being obtained from a parametric integral: $$ p(x|H_i) = \int d\theta \pi(\theta) p(x|H_i;\theta)$$ In this case, the statistical model is Bernoulli trials, with a prior probability on $\theta$, $\pi(\theta)$. Since we're not given any information on what the distribution might be, we can assume $\pi(\theta)=\text{Uniform}(0,1)$ (contrast to the other extreme case in $H_0$, where we have precise prior knowledge of $\theta$: $\pi(\theta) = \delta(\theta-0.5)$).

Once you flip the coin, you can evaluate the likelihood ratio

$$B_{01}(x) = \frac{p(x|H_0)}{\int_0^1 p(x|H_1;\theta)\times 1 d\theta}$$

and also update your posterior probabilities

$$p(H_0|x) = \tfrac{1}{c}0.5\times0.5,$$ $$p(H_1|x) = \tfrac{1}{c}0.5\times\int_0^1 p(x|H_1;\theta)\times1 d\theta,$$

where $c$ is a normalizing constant that only depends on the data $x$ (i.e., the result of the coin toss).

You can also obtain the posterior distribution for $\theta$ under $H_1$, $\pi(\theta|x,H_1)$. At this point, I will describe this for a more general case than observing the outcome of a single coin toss.

Batch Updating

The evolution of this distribution after $n$ coin tosses with $n_h$ heads and $n_t$ tails is

$$\pi(\theta|x=\{n_h ,n_t\}) = \frac{\theta^{n_h-1}(1-\theta)^{n_t-1}}{\text{B} (n_t,n_f)}$$

where the normalizing constant $\text{B}(n_t,n_f)$ is the Beta distribution. This is called the conjugate prior of the binomial distribution, which is itself the distribution for more than one coin toss.

Stray notes

• We were lucky to have the posterior distribution for $\theta$ under $H_1$ be analytically tractable. Not all statistical models have this feature, and require Monte Carlo sampling. See, for instance, this post on CV.
• I have simulated this problem in MATLAB and watched the behaviors of the posteriors. Not surprisingly, the posteriors become more erratic when ground truth is $H_1$ with $\theta$ close to $0.5$.
• It is not entirely clear to me how to deal with the original problem with four hypotheses (above post). But it seems reasonable to start with a broader categorization $p(H_4) = p(H_1\cup H_2 \cup H_3)=0.5$ and then further specifying individual hypotheses in a tree structure.
• The notes from Jim Berger's lectures at CBMS's conference on Model Uncertainty and Multiplicity at UCSC were extremely helpful in clarifying these concepts.
• In light of the fact that I answered my own question, I would like to amend my previous "Thanks for reading" to "Thanks for not downvoting," and I hope (perhaps in vain) that my laziness in attempting to make the internet do my homework for me is atoned for by my diligence in following up on the question and transmitting my findings to the community.