6
$\begingroup$

I am wondering why introductory books on statistics use a conjugate distribution family for the prior instead of using the same pdf of the one we are trying to infer the parameters?

For example, the binomial distribution has the Beta distribution as the prior. They are very similar except for a normalization constant due to the Beta distribution. Why not use another binomial distribution (which can be also uninformative) as the prior?

A case where I see the conjugate prior as necessary is the case when inferring $\Sigma$ of a multivariate Normal given $\mu$ which certainly needs a distribution with symmetrical paramters (e.g. Wishart/Inverse Wishart distribution).

$\endgroup$
8
  • 4
    $\begingroup$ If the observations follow a binomial distribution, the prior is for the success probability. How would you use a binomial distribution as the prior for a probability? $\endgroup$ Commented Sep 15, 2022 at 11:36
  • 5
    $\begingroup$ @Han beta and binomial are not the same, it's not only about the constant. Binomial is discrete and sums to one, beta is continuous and integrates to one, so "changing the constant" would make them improper distributions. $\endgroup$
    – Tim
    Commented Sep 15, 2022 at 11:43
  • 2
    $\begingroup$ The probability spaces are very different. One way to see this is in the binomial the number of successes is the random variable and you are summing over $k$ in $\sum_k {n\choose k}p^k (1-p)^{n-k} = 1$, while in the beta the parameter $p$ is the random variable you are integrating over $p$ in $\int_p \frac{1}{B(\alpha,\beta)}p^\alpha(1-p)^\beta\, dp =1$. But they fit together well as a likelihood and conjugate prior $\endgroup$
    – Henry
    Commented Sep 15, 2022 at 19:43
  • 2
    $\begingroup$ @Han The parameter of the binomial is defined on the unit interval of the real line, not on the values $\left\{0,1,2,...,n\right\}$ nor even on $\left\{0,\frac{1}{n},\frac{2}{n},...,1\right\}$. The prior for $p$ and the pmf $f(x)$ you're using for the data are operating in orthogonal directions on $(p,x)$ $\endgroup$
    – Glen_b
    Commented Sep 16, 2022 at 0:42
  • 1
    $\begingroup$ The likelihood is not a density. $\endgroup$
    – Xi'an
    Commented Sep 16, 2022 at 10:04

3 Answers 3

14
$\begingroup$

For example, the binomial distribution has the Beta distribution as the prior. They are very similar except for a normalization constant due to the Beta distribution. Why not use another binomial distribution (which can be also uninformative) as the prior?

Actually, it's a great counterexample. The beta-binomial model is

$$\begin{align} p &\sim \mathsf{Beta}(\alpha, \beta) \\ X &\sim \mathsf{Bin}(p, n) \end{align}$$

where $X$ is a discrete random variable $X \in \{ 0, 1, 2, 3, \dots , n\}$, and the binomial distribution is parametrized by probability of success $p$ and the sample size $n$. We cannot use binomial distribution as a prior for $p$, because $p$ is continuous and bounded $p \in (0, 1)$. The same applies to many other distributions, for example, a prior for the Poisson's $\lambda$ parameter would be continuous and non-negative, unlike discrete Poisson distribution.

$\endgroup$
2
  • $\begingroup$ I think that the beta-binomial model is the consequence of using a beta as the prior to a binomial model. May I ask if it is a necessary condition to have a continuous distribution (that is also conjugate) as the prior distribution to the (discrete) binomial? $\endgroup$
    – wd violet
    Commented Sep 15, 2022 at 11:47
  • 6
    $\begingroup$ @Han beta is a prior for the parameter $p$, the "probability of success". It is not a prior "for" binomial distribution, but for one of its parameters. $\endgroup$
    – Tim
    Commented Sep 15, 2022 at 11:55
7
$\begingroup$

The nature of your question itself suggests a conceptual misunderstanding.

When we consider a binomial PMF, e.g. $$X \sim \operatorname{Binomial}(n, p)$$ with $$\Pr[X = x] = \binom{n}{x} p^x (1-p)^{n-x}, \quad x \in \{0, 1, \ldots, n\},$$ the support of this random variable is $X \in \{0, 1, 2, \ldots, n\}$. This represents the set of possible elementary outcomes of $X$, and it is in this regard that the sum of the probabilities of all such outcomes equals unity: $$\sum_{x=0}^n \Pr[X = x] = \sum_{x=0}^n \binom{n}{x} p^x (1-p)^{n-x} = (p + (1-p))^n = 1^n = 1$$ by the binomial theorem.

However, for a beta distributed random variable, say $P \sim \operatorname{Beta}(a,b)$, the probability density is $$f_P(p) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} p^{a-1} (1-p)^{b-1}, \quad 0 < p < 1$$ where $a$ and $b$ are shape parameters. Here, $0 < P < 1$ is the support representing the set of elementary outcomes of $P$, and the "sum" of the probabilities of all such outcomes is $$\int_{p=0}^1 f_P(p) \, dp = 1.$$

These distributions do not have the same behavior at all. As other answers have pointed out, it makes no sense to model the (Bayesian) prior $p$ in a binomial likelihood with another binomial distribution because $p$ is a probability, not a count. The notion that $\binom{n}{x}$ and $\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}$ are just "normalization constants" reflects a fundamental misunderstanding that occasionally occurs among students of Bayesian statistics.

The idea behind conjugate distributions really has to do with the concept of the kernel of a probability function. The kernel is the part that depends on the parameter(s), and excludes any multiplicative factors that are constant with respect to those parameter(s). For instance, with respect to the parameter $p$, the kernel of the binomial PMF is

$$\ker(\Pr[X=x]) = p^x (1-p)^{n-x}.$$ The factor $\binom{n}{x}$ is excluded because it does not depend on $p$. The kernel is the basis for the likelihood function with respect to those same parameter(s); e.g., $$\mathcal L(p \mid x) \propto \ker(\Pr[X = x]).$$ The essence of this idea is that a likelihood is a function of the parameter(s) for some observed data $x$; as such, it is only uniquely determined up to a constant of proportionality. For instance, if $X \sim \operatorname{Binomial}(n = 5, p)$ and we observed $X = 3$, we could write the likelihood function of $p$ as $$\mathcal L(p \mid X = 3) = p^3 (1-p)^2, \quad 0 < p < 1,$$ or we could write it as $$\mathcal L(p \mid X = 3) = 157839027384 p^3 (1-p)^2, \quad 0 < p < 1.$$ It doesn't matter because $\mathcal L$ need not satisfy $\int_{p=0}^1 \mathcal L(p) \, dp = 1$. There is a choice that makes this true for general $n, x$, and when we do choose this, we get a beta distribution over $p$. This is why the beta distribution is the conjugate prior for a binomial likelihood.

As another example, if we have a Poisson distributed random variable $\Pr[Y = y] = e^{-\lambda} \lambda^y/y!$ with unknown rate $\lambda$, its kernel with respect to $\lambda$ is $$\ker(\Pr[Y = y]) = e^{-\lambda} \lambda^y, \quad y > 0.$$ So its likelihood is $$\mathcal L(\lambda \mid y) \propto e^{-\lambda} \lambda^y$$ which is proportional to a gamma density with shape $y+1$ and rate $1$; hence the gamma distribution is the conjugate prior for a Poisson likelihood.

$\endgroup$
3
$\begingroup$

While I can't speak for textbook authors, I can think of two reasons why they might choose to do this. Both come from the fact that a conjugate prior will lead to a posterior distribution of a known distribution family.

(1) Simplicity: A well-written textbook should distill the content to the key points, and an example where the posterior does not simplify to a convenient form would add clutter that isn't relevant to discussion. Perhaps the main point could be: the posterior distribution will be a valid pdf or pmf (oh hey, it's a distribution we recognize!)

(2) Computational convenience: With conjugate distributions, algorithms such as Gibbs sampling will be more efficient, as samples can be drawn directly from target distributions without using approximate methods like slice or rejection sampling.

(bonus reason) If the posterior distribution simplifies to a known distribution, closed-form methods can be used rather than MCMC.

In regard to your beta-binomial example, a beta distribution is continuous and bounded on (0,1), which matches the target distribution of the binomial parameter $p$. Using a binomial distribution as a prior for $p$ wouldn't make sense, because its support is discrete, whole-number values up to and including some parameter $n$.

This is by no means a complete answer, and others are welcome to chime in!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.