1
$\begingroup$

A commenter reminds me to be clearer.

In a Bayesian context, the product of a binomial likelihood and a beta prior probability is

$$\left( {\begin{array}{*{20}{c}}n\\x\end{array}} \right)p_{}^x{(1 - p)^{n - x}}\frac{{\Gamma (a + b)}}{{\Gamma (a)\Gamma (b)}}p_{}^{a - 1}{(1 - {p_{}})^{b - 1}} % MathType!MTEF!2!1!+- % feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr % 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq-Jc9 % vqaqpepm0xbba9pwe9Q8fs0-yqaqpepae9pg0FirpepeKkFr0xfr-x % fr-xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 % qadaqadaWdaeaafaqabeGabaaabaWdbiaad6gaa8aabaWdbiaadIha % aaaacaGLOaGaayzkaaGaamiCamaaDaaaleaaaeaacaWG4baaaOGaai % ikaiaaigdacqGHsislcaWGWbGaaiykamaaCaaaleqabaGaamOBaiab % gkHiTiaadIhaaaGcpaWaaSaaaeaacqqHtoWrcaGGOaGaamyyaiabgU % caRiaadkgacaGGPaaabaGaeu4KdCKaaiikaiaadggacaGGPaGaeu4K % dCKaaiikaiaadkgacaGGPaaaa8qacaWGWbWaa0baaSqaaaqaaiaadg % gacqGHsislcaaIXaaaaOGaaiikaiaaigdacqGHsislcaWGWbWaaSba % aSqaaaqabaGccaGGPaWaaWbaaSqabeaacaWGIbGaeyOeI0IaaGymaa % aaaaa!5A9D! $$

Ignoring the normalizing constant, and noting the correspondence between factorials and gamma values for integers, the posterior probability can be written as

$$\begin{array}{c}P\left( {\begin{array}{*{20}{r}}p\end{array}\left| {x,n,a,b} \right.} \right) = \frac{{\Gamma (n)}}{{\Gamma (x)\Gamma (n - x)}}p_{}^x{(1 - p)^{n - x}}\frac{{\Gamma (a + b)}}{{\Gamma (a)\Gamma (b)}}p_{}^{a - 1}{(1 - {p_{}})^{b - 1}}\\\\ = \frac{{\Gamma (n + a + b)}}{{\Gamma (x + a)\Gamma (b + n - x)}}p_{}^{x + a - 1}{(1 - p)^{n - x + b - 1}}\end{array} % MathType!MTEF!2!1!+- % feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr % 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq-Jc9 % vqaqpepm0xbba9pwe9Q8fs0-yqaqpepae9pg0FirpepeKkFr0xfr-x % fr-xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGceaabbeaacaWGqb % WaaeWaaeaafaqaceqabaaabaGaamiCaaaadaabbaqaaiaadIhacaGG % SaGaamOBaiaacYcacaWGHbGaaiilaiaadkgaaiaawEa7aaGaayjkai % aawMcaaiabg2da9maalaaabaGaeu4KdCKaaiikaiaad6gacaGGPaaa % baGaeu4KdCKaaiikaiaadIhacaGGPaGaeu4KdCKaaiikaiaad6gacq % GHsislcaWG4bGaaiykaaaaqaaaaaaaaaWdbiaadchadaqhaaWcbaaa % baGaamiEaaaakiaacIcacaaIXaGaeyOeI0IaamiCaiaacMcadaahaa % Wcbeqaaiaad6gacqGHsislcaWG4baaaOWdamaalaaabaGaeu4KdCKa % aiikaiaadggacqGHRaWkcaWGIbGaaiykaaqaaiabfo5ahjaacIcaca % WGHbGaaiykaiabfo5ahjaacIcacaWGIbGaaiykaaaapeGaamiCamaa % DaaaleaaaeaacaWGHbGaeyOeI0IaaGymaaaakiaacIcacaaIXaGaey % OeI0IaamiCamaaBaaaleaaaeqaaOGaaiykamaaCaaaleqabaGaamOy % aiabgkHiTiaaigdaaaaakeaaaeaapaGaeyypa0ZaaSaaaeaacqqHto % WrcaGGOaGaamOBaiabgUcaRiaadggacqGHRaWkcaWGIbGaaiykaaqa % aiabfo5ahjaacIcacaWG4bGaey4kaSIaamyyaiaacMcacqqHtoWrca % GGOaGaamOyaiabgUcaRiaad6gacqGHsislcaWG4bGaaiykaaaapeGa % amiCamaaDaaaleaaaeaacaWG4bGaey4kaSIaamyyaiabgkHiTiaaig % daaaGccaGGOaGaaGymaiabgkHiTiaadchacaGGPaWaaWbaaSqabeaa % caWGUbGaeyOeI0IaamiEaiabgUcaRiaadkgacqGHsislcaaIXaaaaa % aaaa!9545! $$

what is the reason or better, derivation, for why the gamma coefficient in the second equation has that form? That is, why does one just add the parameter values, n, x, a, b ? One explanation I have seen is that one just recognizes the exponents in the second equation form a beta distribution, and simply forms the associated gamma coefficient with them. That is intuitive, but is there a derivation some where that shows why this is correct?

$\endgroup$
  • 1
    $\begingroup$ I'm afraid that I cannot recognize a "product of distributions" in the expression for $P$. The first (binomial) term clearly is not a distribution--it is only a particular probability (for $x$). (The second (beta) term is indeed the expression of a probability density function for $p$.) $\endgroup$ – whuber May 5 '16 at 19:30
2
$\begingroup$

While the other answers are pretty much right, I think it's somewhat simpler to think about it like this. Bayes' theorem leads to the following:

$$ \require{cancel} \begin{array}{rcl} p(\theta|D) &=& \dfrac{p(D|\theta)p(\theta)}{\int_{\Theta}p(D|\theta)p(\theta)d\theta}\\[2ex] &=& \dfrac{\binom{n}{x}\theta^x(1-\theta)^{n-x}Be(\alpha,\beta)\theta^{\alpha-1}(1-\theta)^{\beta-1}}{\int_\Theta\binom{n}{x}\theta^x(1-\theta)^{n-x}Be(\alpha,\beta)\theta^{\alpha-1}(1-\theta)^{\beta-1}d\theta} \end{array} $$ Since neither the Binomial nor Beta function depend on theta, they can be moved outside the integral in the denominator, so they cancel out and you're left with the typical normalizing Beta function. $$ \require{cancel} \begin{array}{rcl} p(\theta|D) &=& \dfrac{\binom{n}{x}Be(\alpha,\beta)\theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1}}{\binom{n}{x}Be(\alpha,\beta)\int_\Theta\theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1}d\theta}\\[2ex] &=&\dfrac{\cancel{\binom{n}{x}}\cancel{Be(\alpha,\beta)}\theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1}}{\cancel{\binom{n}{x}}\cancel{Be(\alpha,\beta)}\int_\Theta\theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1}d\theta}\\[2ex] &=&\dfrac{\theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1}}{Be(\alpha+x,\beta+n-x)} \end{array} $$

(Incidentally this is why the posterior is unaffected by stopping rules, since as long as the stopping rule doesn't affect the kernel the terms can move outside the integral and cancel)

$\endgroup$
  • $\begingroup$ Thanks Alexander. This is the derivation I needed. With your last result I need just to write the beta in the denominator in terms of its gamma solution. $\endgroup$ – Haynes May 8 '16 at 1:25
  • $\begingroup$ @Haynes Happy to help. $\endgroup$ – Alexander Etz May 8 '16 at 1:34
2
$\begingroup$

I think what you are missing here is the understanding of proportionality. Let $f(x\mid p)$ be the Binomial likelihood and $f(p)$ be the Beta$(a,b)$ prior. Then the posterior $$f(p\mid x) = \dfrac{f(x\mid p)f(p)}{f(x)} \propto f(x \mid p)f(p). $$

Here $f(x)$ is a constant for $p \mid x$, and thus we can ignore it. So when calculating the posterior (without ignoring constants for now), $$f(p \mid x) = \dfrac{1}{f(x)} \dfrac{\Gamma(n) \Gamma(a+b)}{\Gamma(x)\Gamma(n-x)\Gamma(a)\Gamma(b)}p^{x+a-1}(1-p)^{n-x+b-1}.$$

Then using the method mentioned by @RaxYao, you figure out that this is a Beta$(x+a, n-x+b)$ which means that, the constants will equate to the constant for the Beta$(x+a, n-x+b)$. That is,

$$\dfrac{1}{f(x)} \dfrac{\Gamma(n) \Gamma(a+b)}{\Gamma(x)\Gamma(n-x)\Gamma(a)\Gamma(b)} = \dfrac{\Gamma(n+a+b)}{\Gamma(x+a)\Gamma(n-x+b)} $$

So it is not that $\Gamma(x)\Gamma(a) = \Gamma(x+a)$, but in fact,

$$f(x) = \dfrac{\Gamma(x+a)\Gamma(n-x+b) \Gamma(n) \Gamma(a+b)}{\Gamma(n+a+b)\Gamma(x)\Gamma(n-x)\Gamma(a)\Gamma(b)}.$$

$\endgroup$
1
$\begingroup$

The reason is pretty simple. We can compute the coefficient in the following way.

In the Bayesian point of view, the posterior of $p$ is:

$$\begin{align} P\left( {p|{x,n,a,b}} \right) & \propto p^x{(1 - p)^{n - x}}p^{a - 1}{(1 - {p_{}})^{b - 1}} \\ & \propto p_{}^{x + a - 1}{(1 - p)^{n - x + b - 1}} \end{align} $$ Notice that the to make the posterior of $p$ of the form $p_{}^{x + a - 1}{(1 - p)^{n - x + b - 1}} $ and make it to be a proper PDF, i.e., $$ \begin{align} \int P\left( {p|{x,n,a,b}} \right) dp & = 1 \\ \implies \int_p c \times p_{}^{x + a - 1}{(1 - p)^{n - x + b - 1}} & = 1 \end{align} $$ Then it will be very simple to determine the value of $c$. Notice that the Beta distribution with parameter $(x+a, n-x+b)$ can make the last equation feasible. Thus $$c= \frac{\Gamma(x+a+n-x+b)}{\Gamma(x+a)\Gamma(n-x+b)}=\frac{\Gamma(a+n+b)}{\Gamma(x+a)\Gamma(n-x+b)} $$

We say that $p_{}^{x + a - 1}{(1 - p)^{n - x + b - 1}}$ is the kernel of a distribution, which happens to follow the pattern of a Beta distribution.

$\endgroup$
  • 1
    $\begingroup$ Thanks, but that is what I meant by "One explanation I have seen is that one just recognizes the exponents in the second equation form a beta distribution, and simply forms the associated gamma coefficient with them. That is intuitive, but is there a derivation some where that shows why this is correct?" But is there some separate deriviation or argument why the gamma terms can be combined other than recognizing that the final form is beta and combining the parameters as you did? Perhaps not, and your explanation is all that is needed. $\endgroup$ – Haynes May 5 '16 at 20:08
  • $\begingroup$ +1 Ultimately any explanation has to come down to this argument, because the "combination of Gamma terms" is not a mathematical identity about the Gamma function: it arises only because the posterior distribution must be normalized. That exposes what appears to be the fundamental triviality of the question: if Beta$(a,b)^{-1}$ is the normalizing coefficient for a Beta$(a,b)$ distribution, then (merely by plugging in the values) Beta$(a+x,b+n-x)^{-1}$ is the normalizing coefficient for a Beta$(a+x,b+n-x)$ distribution. $\endgroup$ – whuber May 5 '16 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.