33
$\begingroup$

I know that the beta distribution is conjugate to the binomial. But what is the conjugate prior of the beta? Thank you.

$\endgroup$
24
$\begingroup$

It seems that you already gave up on conjugacy. Just for the record, one thing that I've seen people doing (but don't remember exactly where, sorry) is a reparameterization like this. If $X_1,\dots,X_n$ are conditionally iid, given $\alpha,\beta$, such that $X_i\mid\alpha,\beta\sim\mathrm{Beta}(\alpha,\beta)$, remember that $$ \mathbb{E}[X_i\mid\alpha,\beta]=\frac{\alpha}{\alpha+\beta} =: \mu $$ and $$ \mathbb{Var}[X_i\mid\alpha,\beta] = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)} =: \sigma^2 \, . $$ Hence, you may reparameterize the likelihood in terms of $\mu$ and $\sigma^2$ and use as a prior $$ \sigma^2\mid\mu \sim \mathrm{U}[0,\mu(1-\mu)] \qquad \qquad \mu\sim\mathrm{U}[0,1] \, . $$ Now you're ready to compute the posterior and explore it by your favorite computational method.

$\endgroup$
  • 4
    $\begingroup$ No, not MCMC this thing! Quadrature this thing! only 2 parameters - quadrature is the "gold standard" for small dimensional posteriors, both for time and accuracy. $\endgroup$ – probabilityislogic Aug 18 '13 at 22:47
  • 3
    $\begingroup$ Another option is to regard $\psi = \alpha + \beta$ as a measure of precision, and again use $\mu = \frac{\alpha}{\alpha + \beta}$ as an the mean. This is done all the time with Dirichlet processes, and the beta distribution is a special case. So maybe toss a gamma or log-normal prior on $\psi$ and uniform on $\mu$. $\endgroup$ – guy Aug 18 '13 at 23:34
  • 2
    $\begingroup$ To be sure, this isn't conjugate, correct? $\endgroup$ – guy Aug 19 '13 at 16:23
  • 3
    $\begingroup$ Definitely not! $\endgroup$ – Zen Aug 19 '13 at 16:53
  • $\begingroup$ Hi @Zen i'm dealing with this problem right now, but i'm new at Bayesian and im not sure if im understanding the idea. I figured out that you're proposing to find $\int_{0}^{1}{\frac{1}{\mu(1-\mu}}d\mu$ and then use reparametrization, but of course that Was not the idea. Can you please help me to understand? $\endgroup$ – Red Noise Mar 12 '17 at 15:54
23
$\begingroup$

Yes, it has a conjugate prior in the exponential family. Consider the three parameter family $$ \pi(\alpha, \beta \mid a, b, p) \propto \left\{\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\right\}^p \exp\left(a\alpha + b\beta \right). $$ For some values of $(a, b, p)$ this is integrable, although I haven't quite figured out which (I believe $p \ge 0$ and $a < 0, b < 0$ should work - $p = 0$ corresponds to independent exponential distributions so that definitely works, and the conjugate update involves incrementing $p$ so this suggest $p > 0$ works as well).

The problem, and at least part of the reason no one uses it, is that $$ \int_0^\infty \int_0^\infty \left\{\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\right\}^p \exp\left(a\alpha + b\beta \right) = ? $$ i.e. the normalizing constant doesn't have a cloed form.

$\endgroup$
  • $\begingroup$ Ah. That is problematic. I was going to look for an uninformative version of the conjugate prior anyway, so looks like I might as well start with uniform priors over the two parameters. Thanks. $\endgroup$ – Brash Equilibrium Aug 15 '13 at 20:42
  • $\begingroup$ You don't need to normalize it if you're just comparing likelihoods… $\endgroup$ – Neil G Aug 17 '13 at 23:26
  • $\begingroup$ I think you might be missing the action of $p$ in your $\exp$ term as well. It should probably be $pa\alpha$, etc. $\endgroup$ – Neil G Aug 17 '13 at 23:38
  • $\begingroup$ @NeilG $p$ is in the $\exp$, you just have to express things in terms of $\log \Gamma(\cdots)$ instead of $\Gamma(\cdots)$. Doing $pa\alpha$ is just a reparmetrization, it changes nothing. Not sure what you mean "just comparing likelihoods". You can't implement a Gibbs sampler with this prior without using something like Metropolis, which kills the advantage of conditional conjugacy, the normalizing constant depends on $a$ and $b$ which kills putting a prior on them or estimating them by likelihood methods, etc... $\endgroup$ – guy Aug 17 '13 at 23:44
  • 2
    $\begingroup$ @NeilG integral is over $\alpha$ and $\beta$ since those are the random variables. $\endgroup$ – guy Aug 18 '13 at 1:59
9
$\begingroup$

In theory there should be a conjugate prior for the beta distribution. This is because

However the derivation looks difficult, and to quote A Bouchard-Cote's Exponential Families and Conjugate Priors

An important observation to make is that this recipe does not always yields a conjugate prior that is computationally tractable.

Consistent with this, there is no prior for the Beta distribution in D Fink's A Compendium of Conjugate Priors.

$\endgroup$
3
$\begingroup$

I do not believe there is a "standard" (i.e., exponential family) distribution that is the conjugate prior for the beta distribution. However, if one does exist it would have to be a bivariate distribution.

$\endgroup$
  • $\begingroup$ I have no idea about this question, but I did find this handy conjugate prior map that seems to support your answer: johndcook.com/conjugate_prior_diagram.html $\endgroup$ – Justin Bozonier Aug 15 '13 at 3:10
  • $\begingroup$ The conjugate prior is in the exponential family and has three parameters — not two. $\endgroup$ – Neil G Aug 17 '13 at 23:25
  • 1
    $\begingroup$ @Neil, you are definitely right. I guess I should have said it would have to have at least two parameters. $\endgroup$ – user25658 Aug 18 '13 at 18:28
  • $\begingroup$ -1: this answer is clearly wrong in the claim that "conjugate prior does not exist in the exponential family", as is demonstrated in the answer above... $\endgroup$ – Jan Kukacka Apr 24 '18 at 8:34
3
$\begingroup$

Robert and Casella (RC) happen to describe the family of conjugate priors of the beta distribution in Example 3.6 (p 71 - 75) of their book, Introducing Monte Carlo Methods in R, Springer, 2010. However, they quote the result without citing a source.

Added in response to gung's request for details. RC state that for distribution $B(\alpha, \beta)$, the conjugate prior is "... of the form

$$ \pi(\alpha,\beta) \propto \Big\{ \frac{\Gamma(\alpha+\beta)} {\Gamma(\alpha)\Gamma(\beta)} \Big\} ^{\lambda} x_0^{\alpha} y_0^{\beta} $$

where $\{\lambda, x_0, y_0\}$ are hyperparameters, since the posterior is then equal to

$$ \pi(\alpha,\beta \vert x) \propto \Big\{ \frac{\Gamma(\alpha+\beta)} {\Gamma(\alpha)\Gamma(\beta)} \Big\} ^{\lambda} (xx_0)^{\alpha} ((1-x)y_0)^{\beta}." $$

The remainder of the example concerns importance sampling from $\pi(\alpha,\beta \vert x)$ in order to compute the marginal likelihood of $x$.

$\endgroup$
  • 2
    $\begingroup$ I don't have Robert's book available but the posterior is $\pi(\alpha, \beta) \propto \left( \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha) \Gamma(\beta)} \right)^{\lambda +1} (x x_0)^{\alpha-1} \left(y_0 (1-x) \right)^{\beta-1}$. Robert also posted on this topic here mathoverflow.net/questions/20399/… $\endgroup$ – Fred Schoen Jan 13 '16 at 20:22
  • 1
    $\begingroup$ I humbly recommend that the original poster updates the post to indicate that the posterior given in the textbook is incorrect, per Fred Schoen's comment (which is easily verified). $\endgroup$ – RMurphy May 29 '17 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.