# Posterior distribution is impossible depending on which prior hyperparameters are used?

Suppose we randomly select one of two coins and flip it. In that situation we have random variables $$\alpha$$ and $$\delta$$, where $$\alpha$$ tells us which coin we select, and $$\delta$$ tells us whether our coin flip comes up heads.

Suppose we model this as follows: $$\alpha\sim \textrm{Bernoulli}(\rho)$$ $$\beta_0\sim \textrm{Beta}(a_0,a_1)$$ $$\beta_1\sim \textrm{Beta}(b_0,b_1)$$ $$\delta|\alpha,\beta_0,\beta_1\sim \textrm{Bernoulli}(\beta_{\alpha})$$ (So that $$\beta_i$$ is the probability that coin $$i$$ comes up heads.)

I have already worked out that the posterior distribution of $$\alpha$$ given $$\delta$$ is: $$p(\alpha|\delta) = \left(\rho \frac{b_{1-\delta}}{b_0+b_1}\right)^\alpha \left((1-\rho)\frac{a_{1-\delta}}{a_0+b_0}\right)^{1-\alpha}$$ which sure looks like a Bernoulli distribution. But for this to actually be a proper Bernoulli distribution, the two exponentiated terms have to sum to 1, which seems to place very strong restrictions on my choice of hyperparameters $$\rho,a_0,a_1,b_0,b_1$$. In other words, since those two exponentiated terms have to sum to 1, we can derive: $$\rho = \frac{\left(1-\frac{a_{1-\delta}}{a_0+a_1}\right)}{\left(\frac{b_{1-\delta}}{b_0+b_1}-\frac{a_{1-\delta}}{a_0+a_1}\right)}$$ which seems to be a very restrictive constraint! I had thought that I could make any choice for setting my hyperparameters and still derive a proper posterior distribution. Was I wrong about that? For this very simple model, are many choices of prior hyperparameters $$\rho,a_0,a_1,b_0,b_1$$ actually impossible? (And if so, why? It certainly seems like my prior probability of choosing either coin should not depend on the biases of the two coins, but per my derivation it appears that it does!)

• The posterior probabilities should sum to one by definition, so there must be a mistake in your calculation. If you will detail your steps, we might be able to point out where you went wrong May 4 at 15:34

Let's work through the steps. To begin with we have $$$$p(\delta|\beta)\,p(\beta|\alpha)\,p(\alpha) ,$$$$ where \begin{align} p(\delta|\beta) &= \textsf{Bernoulli}(\delta|\beta) \\ p(\beta|\alpha) &= \textsf{Beta}(\beta|a_\alpha,b_\alpha) \\ p(\alpha) &= \textsf{Bernoulli}(\alpha|\rho) . \end{align} Note that instead of indexing the parameter $$\beta$$ with $$\alpha$$ as in the question, I have indexed the hyperparameters $$(a_\alpha,b_\alpha)$$ of the beta distribution with $$\alpha$$.

Then $$$$p(\delta|\alpha) = \int p(\delta|\beta)\,p(\beta|\alpha)\,d\beta = \textsf{Bernoulli}\Big(\delta\,\big|\,\frac{a_\alpha}{a_\alpha+b_\alpha}\Big) .$$$$ Consequently, $$$$p(\alpha|\delta) = \frac{p(\delta|\alpha)\,p(\alpha)}{p(\delta)} , %= \textsf{Bernoulli} \Big(\delta\,\big|\,\frac{a_\alpha}{a_\alpha+b_\alpha}\Big)\,\textsf{Bernoulli}(\alpha|\rho)$$$$ where $$$$p(\delta) = \sum_{\alpha\in\{0,1\}} p(\delta|\alpha)\,p(\alpha) .$$$$ Dividing by $$p(\delta)$$ guarantees the probabilities for $$\alpha$$ sum to one without any special restrictions on the hyperparameters.

Here is some additional detail: $$$$p(\delta|\alpha)\,p(\alpha) = \left(\frac{a_\alpha}{a_\alpha+b_\alpha}\right)^\delta \left(\frac{b_\alpha}{a_\alpha+b_\alpha}\right)^{1-\delta} \rho^\alpha\,(1-\rho)^{1-\alpha} .$$$$

• I follow your reasoning here, but it leaves me not understanding where or whether I've gone wrong in the reasoning in my post. $p(\alpha|\delta)$ does have the distribution I derived, doesn't it? And that distribution is only a proper Bernoulli for heavily constrained values of my hyperparameters -- right?
– Ceph
May 4 at 15:15
• As I tried to show, there are no restrictions on the hyperparameters (beyond $0<\rho<1$ and $a_\alpha,b_\alpha>0$). That means your posterior is wrong.
– mef
May 4 at 17:06
• Let me ask the question this way -- in your derivation is the posterior $p(\alpha|\delta)$ a Bernoulli distribution, and if so, what is the mean of that distribution?
– Ceph
May 4 at 17:24
• Since $\alpha \in \{0,1\}$, it must have a Bernoulli distribution. I've added some additional detail to my answer. Perhaps you can work the rest out.
– mef
May 4 at 17:59

The distribution you are describing is a mixture of Bernoulli distributions with the likelihood function

$$p(\delta|\rho,\beta_0,\beta_1) = \rho \, \beta_0^{\delta} (1-\beta_0)^{1-\delta} + (1 - \rho) \, \beta_1^{\delta} (1-\beta_1)^{1-\delta}$$

It does not have a closed-form solution, you need to use an optimization algorithm like E-M for maximum a posteriori solution, MCMC for Monte Carlo approximation of the posterior, or other approach.

• Thank you -- but the posterior distribution I am interested in is $p(\alpha|\delta)$. That one doesn't appear to have the form of a mixture of Bernoullis, does it?
– Ceph
May 4 at 14:20