2
$\begingroup$

Firstly, I apologize if this question is trivial and if my use of notation/terminology is not on point. I am not a statistician/mathematician but have been taken a statistics course to be able to carry out a project. In the course, we define a lot of different distributions, but there is no rationale for why a distribution is expressed the way it is so I find it difficult to understand. Currently, I am trying to understand the derivation of the Beta distribution as a distribution modeling the unknown probability in data following (drawn from?) binomial distribution.

I attempted to show myself that if X~Binomial(p,n), and p~U(0,1), then P(p|X)~Beta but turns out I don't have the skill set to do this derivation. I am familiar with the Bayes rule. In this case, I have the likelihood for the data and the prior for the probabilities (since I don't know anything about it, except that it's a probability and therefore needs to be between 0 and 1, Im assuming a uniform distribution). Then, following Bayes rule, what I would do to get to the posterior for the probability is to multiply the likelihood with the prior and divide it by the marginalization of this multiplication.

So I started like this:

The likelihood: $$f_{(X|p)}(X) = {n \choose x} p^{x} (1-p) ^{(n-x)}$$

The prior: $$f_{P}(p) = \frac{1}{(max - min)} = \frac{1}{1-0} = 1$$

Likelihood * prior (the joint distribution): $$f_{XP}(x,p)= f_{X|P=p}(x) f_{P}(p) $$ $$f_{XP}(x,p)= {n \choose x} p^{x} (1-p) ^{(n-x)} \times 1 $$ $$f_{XP}(x,p)= {n \choose x} p^{x} (1-p) ^{(n-x)}$$

marginalization of the joint distribution: $$ \int {n \choose x} p^{x} (1-p) ^{(n-x)} \,dx $$

Plugging into Bayes rule to get the posterior for p:

$$f_{P|X}(p)= \frac{{n \choose x} p^{x} (1-p) ^{(n-x)}}{\int {n \choose x} p^{x} (1-p) ^{(n-x)} \,dx} $$

How do I get to Beta distribution from here? I don't see how this yields the Beta distribution. If this is not the way to go, could someone help me figure out how to derive Beta in the context of Binomial distribution and help me prove to myself that the posterior of p would indeed be Beta distributed if X is binomially distributed and I assume a uniform prior for p?

I have found plenty of questions here asking about the relationship between binomial and beta distributions and I get the relationship. Binomial distribution models the number of "successes" while the Beta distribution models a probability distribution over the success probability.However, I feel uncomfortable just accepting the expression + definition of Beta and moving on without proving this relationship to myself and the previous questions I found here have not answered this specific question. So, while there seem to be a lot of questions here about Beta, I do not think this is a duplicate.

Thank you for your help in advance!

$\endgroup$
2
  • $\begingroup$ Perhaps the key to completing this is to recognize that the uniform distribution is a beta distribution and to use Bayes' Theorem. See my Answer for an outline. $\endgroup$
    – BruceET
    May 4 at 5:51
  • $\begingroup$ Thomas Bayes famously illustrated this in 1764 with a description of rolling balls on a pool table: search for it! $\endgroup$
    – whuber
    May 4 at 12:44
2
$\begingroup$

On the parameter $p$ you have the prior distribution $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(\alpha_0=1, \beta_0=1)$ with density function $$f_p(p) = p^0(1-p)^0 = p^{\alpha_0-1}(1-p)^{\beta_0-1},$$ for $0 < p < 1,$ where the symbol $\propto$ (pronounced 'proportional to') indicates that we have omitted the norming constant factor that makes $f_p(p)$ integrate to $1$ over $(0,1).$

Based on $n$ Bernoulli trials with $x$ Successes among them, we have the likelihood function $$f_{x|p}(x|p) \propto p^x(1-p)^{n-x}.$$

Then by Bayes' Theorem that POSTERIOR $\propto$ PRIOR $\times$ LIKELIHOOD. we have the posterior distribution with density function $$f_{p|x}(p|x) \propto p^{\alpha_0-1}(1-p)^{\beta_0-1} \times p^x(1-p)^{n-x}\\ \propto p^{\alpha_0+x-1}(1-p)^{\beta_0+n-x-1}\\ \propto p^{\alpha_n -1}(1-p)^{\beta_n - 1},$$ where $\alpha_n = \alpha_0 + x = 1+x$ and $\beta_n = \beta_0+n-x = 1+n-x.$

Because the beta prior and the binomial likelihood are conjugate (mathematically compatible), we can recognize the final term of the relationship above as the kernel (density without norming constant) of the distribution $\mathsf{Beta}(\alpha_n, \beta_n).$

There is only one possible norming constant for this posterior beta kernel, so we can be sure we have the correct posterior distribution.

If you want to supply the norming constants (with $Gamma$ functions and binomial coefficients) throughout, then you can express factorials as $Gamma$ functions to find that the integral in the denominator of the final displayed relationship of your Question simplifies to the norming constant of $\mathsf{Beta}(\alpha_n, \beta_n).$

Notes: (1) In Bayesian statistics one of the attractions of using a conjugate prior and likelihood is that conjugacy often makes the posterior distribution obvious.

(2) Even in frequentist arguments, likelihoods are often defined only 'up to a constant multiple'. The norming constant is irrelevant if the task is to maximize the likelihood.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.