We know, for example, that the conjugate relationship between the classic beta-binomial is as follows:

\begin{align} y &∼ \mathcal{Bin}(n,\ θ) \\ θ &∼ \mathcal{Beta}(α,\ β) \\ θ|y &∼ \mathcal{Beta}(y + α,\ n − y + β) \end{align}

Notice how the posterior is just a change in parameterization compared to the prior. Is the posterior of conjugate priors just a change in parameters?

  • 2
    $\begingroup$ How do you define the concept of conjugate priors (if not this)? $\endgroup$ – Juho Kokkala Apr 1 '16 at 11:53
  • 1
    $\begingroup$ $Beta(y+\alpha, n-y+\beta)$ is not "the same" as $Beta(\alpha,\beta)$, so as far as I understand your comment, the classic beta-binomial would not be conjugate, either. (So I apparently don't understand what you mean by 'the same') $\endgroup$ – Juho Kokkala Apr 1 '16 at 12:03
  • 3
    $\begingroup$ Please learn how to use math typesetting. meta.math.stackexchange.com/questions/5020/… $\endgroup$ – Sycorax Apr 1 '16 at 13:10
  • 1
    $\begingroup$ I am not sure that the title of the question matches with the actual question. Maybe something like "Does conjugate priors just lead to a posterior being a modification of parameters of the prior?" ? $\endgroup$ – peuhp Apr 1 '16 at 13:57
  • 4
    $\begingroup$ Given that there is an upvoted & accepted answer to this Q, it isn't clear that it is too unclear to be answered. $\endgroup$ – gung - Reinstate Monica Apr 1 '16 at 15:30

This question is actually somewhat subtle, and it brings to attention an interesting quirk of usage that I hadn't noticed before.

For every practical definition of conjugate distributions that I'm familiar with, it is the case that the posterior of a model using a conjugate prior is a modified form of the prior. The wikipedia definition follows the "practicality" (convenience) convention, for example:

In Bayesian probability theory, if the posterior distributions $p(\theta|x)$ are in the same family as the prior probability distribution $p(\theta)$, the prior and posterior are then called conjugate distributions, and the prior is called a conjugate prior for the likelihood function

However, a distinction can be found in the formal definition of conjugacy in Gelman's Bayesian Data Analysis, 3rd edition, p. 35:

If $\mathcal{F}$ is a class of sampling distributions $p(y|\theta)$ and $\mathcal{P}$ is a class of prior distributions for $\theta$, then the class $\mathcal{P}$ is conjugate for $\mathcal{F}$ if $$ p(\theta|y)\in\mathcal{P}\forall p(\cdot|\theta)\in\mathcal{F} \text{ and } p(\cdot)\in\mathcal{P}. $$ This definition is formally vague since if we choose $\mathcal{P}$ as the class of all distributions, then $\mathcal{P}$ is always conjugate no matter what class of sampling distribution is used.

Obviously the construction in the final sentence has little practical utility: if all distributions are conjugate, then the distinction between conjugate and non-conjugate distributions is trivial. Instead, it is common to take $\mathcal{P}$ to be the set of all densities having the same functional form of the likelihood, giving rise to the practical convenience properties of conjugacy, namely that the posterior is the form of the prior.

  • 1
    $\begingroup$ (+1) Wonder if there's a formal definition of what would be practically useful conjugacy then. Perhaps to do with having a sufficient statistic that doesn't grow with the sample size, so the posteriors don't keep getting more complicated than the priors when you update. $\endgroup$ – Scortchi - Reinstate Monica Apr 1 '16 at 14:03
  • $\begingroup$ @Scortchi I wonder the same thing. I was actually surprised when I looked up the definition to answer this question. At a minimum, the usage is imprecise, but every Bayesian worth a Gibbs sampler knows that conjugacy is purely jargon to indicate convenience. $\endgroup$ – Sycorax Apr 1 '16 at 14:07
  • 1
    $\begingroup$ Related: Aside from the exponential family, where else can conjugate priors come from?. (I should do what I said I was going to do & ask a question about it. Can't just be coincidence that the useful conjugates are nearly all from the exponential family, & have fixed-dimensional sufficient statistics when they're not.) $\endgroup$ – Scortchi - Reinstate Monica Apr 1 '16 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy