46
$\begingroup$

I know that the beta distribution is conjugate to the binomial. But what is the conjugate prior of the beta? Thank you.

$\endgroup$
2

5 Answers 5

32
$\begingroup$

It seems that you already gave up on conjugacy. Just for the record, one thing that I've seen people doing (but don't remember exactly where, sorry) is a reparameterization like this. If $X_1,\dots,X_n$ are conditionally iid, given $\alpha,\beta$, such that $X_i\mid\alpha,\beta\sim\mathrm{Beta}(\alpha,\beta)$, remember that $$ \mathbb{E}[X_i\mid\alpha,\beta]=\frac{\alpha}{\alpha+\beta} =: \mu $$ and $$ \mathbb{Var}[X_i\mid\alpha,\beta] = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)} =: \sigma^2 \, . $$ Hence, you may reparameterize the likelihood in terms of $\mu$ and $\sigma^2$ and use as a prior $$ \sigma^2\mid\mu \sim \mathrm{U}[0,\mu(1-\mu)] \qquad \qquad \mu\sim\mathrm{U}[0,1] \, . $$ Now you're ready to compute the posterior and explore it by your favorite computational method.

$\endgroup$
10
  • 6
    $\begingroup$ No, not MCMC this thing! Quadrature this thing! only 2 parameters - quadrature is the "gold standard" for small dimensional posteriors, both for time and accuracy. $\endgroup$ Commented Aug 18, 2013 at 22:47
  • 5
    $\begingroup$ Another option is to regard $\psi = \alpha + \beta$ as a measure of precision, and again use $\mu = \frac{\alpha}{\alpha + \beta}$ as an the mean. This is done all the time with Dirichlet processes, and the beta distribution is a special case. So maybe toss a gamma or log-normal prior on $\psi$ and uniform on $\mu$. $\endgroup$
    – guy
    Commented Aug 18, 2013 at 23:34
  • 3
    $\begingroup$ To be sure, this isn't conjugate, correct? $\endgroup$
    – guy
    Commented Aug 19, 2013 at 16:23
  • 3
    $\begingroup$ Definitely not! $\endgroup$
    – Zen
    Commented Aug 19, 2013 at 16:53
  • 1
    $\begingroup$ Another option is to parameterize by "concentration", $c = \sqrt{\alpha^2 + \beta^2}$, which scales independent of the mean. Velten et al., 2015 use this with uniform prior on the mean and log-normal prior on $c$. Details are in their supplement (PDF). $\endgroup$
    – merv
    Commented Jul 6, 2018 at 22:30
33
$\begingroup$

Yes, it has a conjugate prior in the exponential family. Consider the three parameter family $$ \pi(\alpha, \beta \mid a, b, p) \propto \left\{\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\right\}^p \exp\left(a\alpha + b\beta \right). $$ For some values of $(a, b, p)$, this is integrable, although I haven't quite figured out which ones (I believe $p \ge 0$ and $a < 0, b < 0$ should work — $p = 0$ corresponds to independent exponential distributions so that definitely works; moreover, the conjugate update involves incrementing $p$ so this suggests that $p > 0$ works as well).

The problem, and at least part of the reason no one uses it, is that $$ \int_0^\infty \int_0^\infty \left\{\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\right\}^p \exp\left(a\alpha + b\beta \right) = ? $$ i.e. the normalizing constant doesn't have a closed form.

$\endgroup$
9
  • $\begingroup$ Ah. That is problematic. I was going to look for an uninformative version of the conjugate prior anyway, so looks like I might as well start with uniform priors over the two parameters. Thanks. $\endgroup$ Commented Aug 15, 2013 at 20:42
  • 1
    $\begingroup$ You don't need to normalize it if you're just comparing likelihoods… $\endgroup$
    – Neil G
    Commented Aug 17, 2013 at 23:26
  • $\begingroup$ I think you might be missing the action of $p$ in your $\exp$ term as well. It should probably be $pa\alpha$, etc. $\endgroup$
    – Neil G
    Commented Aug 17, 2013 at 23:38
  • $\begingroup$ @NeilG $p$ is in the $\exp$, you just have to express things in terms of $\log \Gamma(\cdots)$ instead of $\Gamma(\cdots)$. Doing $pa\alpha$ is just a reparmetrization, it changes nothing. Not sure what you mean "just comparing likelihoods". You can't implement a Gibbs sampler with this prior without using something like Metropolis, which kills the advantage of conditional conjugacy, the normalizing constant depends on $a$ and $b$ which kills putting a prior on them or estimating them by likelihood methods, etc... $\endgroup$
    – guy
    Commented Aug 17, 2013 at 23:44
  • 2
    $\begingroup$ @NeilG integral is over $\alpha$ and $\beta$ since those are the random variables. $\endgroup$
    – guy
    Commented Aug 18, 2013 at 1:59
11
$\begingroup$

In theory there should be a conjugate prior for the beta distribution. This is because

However the derivation looks difficult, and to quote A Bouchard-Cote's Exponential Families and Conjugate Priors

An important observation to make is that this recipe does not always yields a conjugate prior that is computationally tractable.

Consistent with this, there is no prior for the Beta distribution in D Fink's A Compendium of Conjugate Priors.

$\endgroup$
1
6
$\begingroup$

Robert and Casella (RC) happen to describe the family of conjugate priors of the beta distribution in Example 3.6 (p 71 - 75) of their book, Introducing Monte Carlo Methods in R, Springer, 2010. However, they quote the result without citing a source.

Added in response to gung's request for details. RC state that for distribution $B(\alpha, \beta)$, the conjugate prior is "... of the form

$$ \pi(\alpha,\beta) \propto \Big\{ \frac{\Gamma(\alpha+\beta)} {\Gamma(\alpha)\Gamma(\beta)} \Big\} ^{\lambda} x_0^{\alpha} y_0^{\beta} $$

where $\{\lambda, x_0, y_0\}$ are hyperparameters, since the posterior is then equal to

$$ \pi(\alpha,\beta \vert x) \propto \Big\{ \frac{\Gamma(\alpha+\beta)} {\Gamma(\alpha)\Gamma(\beta)} \Big\} ^{\lambda} (xx_0)^{\alpha} ((1-x)y_0)^{\beta}." $$

The remainder of the example concerns importance sampling from $\pi(\alpha,\beta \vert x)$ in order to compute the marginal likelihood of $x$.

$\endgroup$
2
  • 3
    $\begingroup$ I don't have Robert's book available but the posterior is $\pi(\alpha, \beta) \propto \left( \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha) \Gamma(\beta)} \right)^{\lambda +1} (x x_0)^{\alpha-1} \left(y_0 (1-x) \right)^{\beta-1}$. Robert also posted on this topic here mathoverflow.net/questions/20399/… $\endgroup$ Commented Jan 13, 2016 at 20:22
  • 2
    $\begingroup$ I humbly recommend that the original poster updates the post to indicate that the posterior given in the textbook is incorrect, per Fred Schoen's comment (which is easily verified). $\endgroup$
    – RMurphy
    Commented May 29, 2017 at 21:04
2
$\begingroup$

I do not believe there is a "standard" (i.e., exponential family) distribution that is the conjugate prior for the beta distribution. However, if one does exist it would have to be a bivariate distribution.

$\endgroup$
4
  • $\begingroup$ I have no idea about this question, but I did find this handy conjugate prior map that seems to support your answer: johndcook.com/conjugate_prior_diagram.html $\endgroup$ Commented Aug 15, 2013 at 3:10
  • 1
    $\begingroup$ The conjugate prior is in the exponential family and has three parameters — not two. $\endgroup$
    – Neil G
    Commented Aug 17, 2013 at 23:25
  • 1
    $\begingroup$ @Neil, you are definitely right. I guess I should have said it would have to have at least two parameters. $\endgroup$
    – user25658
    Commented Aug 18, 2013 at 18:28
  • 1
    $\begingroup$ -1: this answer is clearly wrong in the claim that "conjugate prior does not exist in the exponential family", as is demonstrated in the answer above... $\endgroup$ Commented Apr 24, 2018 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.