Understanding the Beta conjugate prior in Bayesian inference about a frequency

Question

Following is an excerpt from Bolstad's Introduction to Bayesian Statistics.

For all you experts out there, this might be trivial but I don't understand how the author concludes that we don't have to do any integration to calculate the posterior probability for some value of $\pi$. I understand the second expression which is the proportionality and where all the terms came from( likelihood x Prior ). Furthermore, I understand,we don't have to worry about the denominator since only the numerator is directly proportional. But moving on to the third equation, aren't we forgetting about the denominator of the Bayes Rule? Where did it go ? And the value computed by the Gamma functions, isn't that a constant ? Doesn't constants cancel out in the Bayes theorem ?

Answer 1

The point is that we know what the posterior is proportional to and it so happens that we do not need to do the integration to get the (constant) denominator, because we recognise that a distribution with probability density function proportional to $x^{\alpha-1} \times (1-x)^{\beta-1}$ (such as the posterior) is a beta distribution. Since the normalizing constant for such a beta pdf is $\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}$, we get the posterior pdf without integration. And yes, the normalizing constant in Bayes theorem is a constant (given the observed data and the prior assumed) just like the normalizing constant for the posterior density.

Answer 2

The setup

You have this model: \begin{align*} p & \, \sim \, \text{beta}(\alpha, \beta) \\ x \, | \, p & \, \sim \, \text{binomial}(n, p) \end{align*} The densities for which are \begin{equation*} f(p) = \frac{1}{B(\alpha, \beta)} p^{\alpha - 1} (1 - p)^{\beta - 1} \end{equation*} \begin{equation*} g(x \, | \, p) = {n \choose x} p^x (1 - p)^{n - x} \end{equation*} and in particular note that \begin{equation*} \frac{1}{B(\alpha, \beta)} = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}. \end{equation*}

The implicit version

Now. The posterior distribution is proportional to the prior $f$ multiplied by the likelihood $g$. We can ignore constants (i.e. things that aren't $p$), yielding: \begin{align*} h(p \, | \, x) & \propto f(p) g(p \, | \, x) \\ & = p^{\alpha - 1} (1 - p)^{\beta - 1} p^x p^{n - x} \\ & = p^{\alpha + x - 1} (1 - p)^{\beta + n - x - 1}. \end{align*}

This has the 'shape' of a beta distribution with parameters $\alpha + x$ and $\beta + n - x$, and we know what the corresponding normalizing constant for a beta distribution with those parameters should be: $1 / B(\alpha + x, \beta + n - x)$. Or, in terms of gamma functions, \begin{equation*} \frac{1}{B(\alpha + x, \beta + n - x)} = \frac{\Gamma(n + \alpha + \beta)}{\Gamma(\alpha + x)\Gamma(\beta + n - x)}. \end{equation*} In other words we can do a bit better than a proportional relation without any extra legwork, and go straight to equality: \begin{equation*} h(p \, | \, x) = \frac{\Gamma(n + \alpha + \beta)}{\Gamma(\alpha + x)\Gamma(\beta + n - x)} p^{\alpha + x - 1} (1 - p)^{\beta + n - x - 1}. \end{equation*}

So one can use knowledge of the structure of a beta distribution to easily recover an expression for the posterior, rather than going through some messy integration and the like.

It sort of gets around to the full posterior by implicitly cancelling the normalizing constants of the joint distribution, which can be confusing.

The explicit version

You could also grind things out procedurally, which can be clearer.

It's not actually all that much longer. Note that we can express the joint distribution as \begin{align*} f(p)g(x \, | \, p) = \frac{1}{B(\alpha, \beta)}{n \choose x} p^{\alpha + x - 1} (1 - p)^{\beta + n - x - 1} \end{align*} and the marginal distribution of $x$ as \begin{align*} \int_{0}^{1}f(p)g(x \, | \, p)dp & = \frac{1}{B(\alpha, \beta)}{n \choose x} \int_{0}^{1} p^{\alpha + x - 1} (1 - p)^{\beta + n - x - 1} dp \\ & = \frac{1}{B(\alpha, \beta)}{n \choose x} \frac{\Gamma(\alpha + x)\Gamma(\beta + n - x)}{\Gamma(\alpha + \beta + n - x)} \end{align*}

So we can express the posterior using Bayes' theorem by \begin{align*} h(p \, | \, x) & = \frac{f(p) g(x \, | \, p)}{\int_{0}^{1}f(p) g(x \, | \, p)dp} \\ & = \frac{\frac{1}{B(\alpha, \beta)}{n \choose x} p^{\alpha + x - 1} (1 - p)^{\beta + n - x - 1}}{\frac{1}{B(\alpha, \beta)}{n \choose x} \frac{\Gamma(\alpha + x)\Gamma(\beta + n - x)}{\Gamma(\alpha + \beta + n)}} \\ & = \frac{\Gamma(n + \alpha + \beta)}{\Gamma(\alpha + x)\Gamma(\beta + n - x)} p^{\alpha + x - 1} (1 - p)^{\beta + n - x - 1} \end{align*} which is the same thing we got previously.

Answer 3

General Remarks

To make the answer given by @Björn a bit more explicit and in the same time more general, we should remember that we arrived at the Bayes Theorem from

$p(\theta|X) \times p(X) = p(X,\theta)=p(X|\theta)\times p(\theta)$

$\implies p(\theta|X) = \frac{p(X|\theta)\times p(\theta)}{p(X)}$ (Bayes Thereom)

where $X$ represents the observed data and $\theta$ our unknown parameter we would like to make probabilistic inferences about -- in the question's case the parameter is an unknown frequency $\pi$. Let's not worry for now whether we are talking about vectors or scalars to keep it simple.

Marginalization in the continuous case leads to

$p(X) = \int_{-\infty}^{+\infty}{p(X,\theta)d\theta}=\int_{-\infty}^{+\infty}{p(X|\theta)\times p(\theta)d\theta}$

where the joint distribution $p(X,\theta)$ equals $likelihood \times prior$ as we have seen above. It is a constant since after 'integrating out' the parameter it only depends on constant terms.

Therefore we can reformulate the Bayes Theorem as

$p(\theta|X) = Const. \times p(X|\theta)\times p(\theta)$ with $Const. = \frac{1}{p(X)} = \frac{1}{\int{p(X|\theta)\times p(\theta)d\theta}}$

and thus arrive at the usual proportionality form of Bayes Theorem.

Application to the problem a hand

Now we are ready to simply plug in what we know since $likelihood \times prior$ in the question's case is of the form

$p(X,\theta)= p(X|\theta)\times p(\theta) = A \cdot \theta^{\,a + y - 1}(1-\theta)^{b + n - y - 1} = A\cdot \theta^{\,a' - 1}(1-\theta)^{b' - 1}$

where $a' = a+y$, $b' = b+n-y$ and where $A = \frac{1}{B(a,b)}\binom{n}{y}$ collects the constant terms from the binomial likelihood and the beta prior.

We can now use the answer given by @Björn to find that this integrates to the Beta function $B(a',b')$ times the collection of constant terms $A$ so that

$p(X) = A\cdot\int_0^1{\theta^{\,a' - 1}(1-\theta)^{b' - 1}d\theta}=A\cdot B(a',b')$

$\implies p(\theta|X) = \frac{A\cdot\theta^{\,a' - 1}(1-\theta)^{b' - 1}}{A\cdot B(a',b')}=\frac{\theta^{\,a' - 1}(1-\theta)^{b' - 1}}{B(a',b')}$

Note, that any constant term in the joint distribution will allways cancel out, since it will appear in the nominator and the denominator at the same time (cf. the answer given by @jtobin) so we really do not have to bother.

Thus we recognize that our posterior distribution is in fact a beta distribution where we can simply update the prior's parameters $a' = a+y$ and $b' = b+n-y$ to arrive at the posterior. This is why the beta distributed prior is called a conjugate prior.