34
$\begingroup$

I have been trying to understand the idea of conjugate priors in Bayesian statistics for a while but I simply don't get it. Can anyone explain the idea in the simplest possible terms, perhaps using the "Gaussian prior" as an example?

$\endgroup$

4 Answers 4

29
$\begingroup$

A prior for a parameter will almost always have some specific functional form (written in terms of the density, generally). Let's say we restrict ourselves to one particular family of distributions, in which case choosing our prior reduces to choosing the parameters of that family.

For example, consider a normal model $Y_i \stackrel{_\text{iid}}{\sim} N(\mu,\sigma^2)$. For simplicity, let's also take $\sigma^2$ as known. This part of the model - the model for the data - determines the likelihood function.

To complete our Bayesian model, here we need a prior for $\mu$.

As mentioned above, commonly we might specify some distributional family for our prior for $\mu$ and then we only have to choose the parameters of that distribution (for example, often prior information may be fairly vague - like roughly where we want the probability to concentrate - rather than of very specific functional form, and we may have enough freedom to model what we want by choosing the parameters - say to match a prior mean and variance).

If it turns out that the posterior for $\mu$ is from the same family as the prior, then that prior is said to be "conjugate".

(What makes it turn out to be conjugate is the way it combines with the likelihood)

So in this case, let's take a Gaussian prior for $\mu$ (say $\mu\sim N(\theta,\tau^2)$). If we do that, we see that the posterior for $\mu$ is also Gaussian. Consequently, the Gaussian prior was a conjugate prior for our model above.

That's all there is to it really -- if the posterior is from the same family as the prior, it's a conjugate prior.

In simple cases you can identify a conjugate prior by inspection of the likelihood. For example, consider a binomial likelihood; dropping the constants, it looks like a beta density in $p$; and because of the way powers of $p$ and $(1-p)$ each combine, it will multiply by a beta prior to also give a product of powers of $p$ and $(1-p)$ ... so we can see immediately from the likelihood that the beta will be a conjugate prior for $p$ in the binomial likelihood.

In the Gaussian case it's easiest to see that it will happen by considering the log-densities and the log-likelihood; the log-likelihood will be quadratic in $\mu$ and the sum of two quadratics is quadratic, so a quadratic log-prior + quadratic log-likelihood gives a quadratic posterior (each of the coefficients of the highest order term will of course be negative).

$\endgroup$
12
$\begingroup$

If your model belongs to an exponential family, that is, if the density of the distribution is of the form $$f(x|\theta)=h(x)\exp\{T(\theta)\cdot S(x)-\psi(\theta)\}\qquad x\in\mathcal{X}\quad\theta\in\Theta$$ with respect to a given dominating measure (Lebesgue, counting, &tc.), where $t\cdot s$ denotes a scalar product over $\mathbb{R}^d$ and $$T:\mathcal{X}\longrightarrow \mathbb{R}^d\qquad S:\Theta\longrightarrow \mathbb{R}^d$$ are measurable functions, the conjugate priors on $\theta$ are defined by densities of the form $$\pi(\theta|\xi,\lambda)=C(\xi,\lambda)\exp\{T(\theta)\cdot \xi-\lambda\psi(\theta)\}$$ [with respect to an arbitrarily-chosen dominating measure $\text{d}\nu$ on $\Theta$] with $$C(\xi,\lambda)^{-1}=\int_\Theta \exp\{T(\theta)\cdot \xi-\lambda\psi(\theta)\} \text{d}\nu<\infty$$ and $\lambda\in\Lambda\subset\mathbb{R}_+$, $\xi\in\Xi\subset \lambda T(\mathcal{X})$

The choice of the dominating measure is determinantal for the family of priors. If for instance one faces a Normal mean likelihood on $\mu$ as in Glen_b's answer, choosing the Lebesgue measure $\text{d}\mu$ as the dominating measure leads to Normal priors being conjugate. If instead one chooses $(1+\mu^2)^{-2}\text{d}\mu$ as the dominating measure, the conjugate priors are within the family of distributions with densities $$\exp\{-\alpha(\mu-\mu_0)^2\} \qquad\alpha>0,\ \ \mu_0\in\mathbb R$$ with respect to this dominating measure, and are thus no longer Normal priors. This difficulty is essentially the same as the one of choosing a particular parameterisation of the likelihood and opting for the Lebesgue measure for this parameterisation. When faced with a likelihood function, there is no inherent (or intrinsic or reference) dominating measure on the parameter space.

Outside this exponential family setting, there is no non-trivial family of distributions with a fixed support that allows for conjugate priors. This is a consequence of the Darmois-Pitman-Koopman lemma.

$\endgroup$
10
  • 15
    $\begingroup$ "in simplest possible terms?" Perhaps an explanation that does not assume prior knowledge of measures would be more useful to the OP. $\endgroup$
    – user44764
    Oct 13, 2015 at 20:21
  • 4
    $\begingroup$ alas, I am afraid conjugate priors are meaningless without a measure background (even though this is the best kept secret in the Universe). $\endgroup$
    – Xi'an
    Oct 13, 2015 at 20:50
  • 9
    $\begingroup$ In my opinion, "simplest possible terms" is open to interpretation, and an explanation which makes use of advanced math such as measure theory might still be "simple" in some sense, maybe even "simpler" than an explanation which avoids such machinery. In any case, such an explanation might be very enlightening to someone who has the necessary background to understand it, and it is harmless to include an answer like this in a list of various ways to explain a topic. We write answers not just for OP but for all future readers. $\endgroup$
    – littleO
    Jan 2, 2020 at 12:48
  • 2
    $\begingroup$ @LBogaardt Your criticism would carry more weight if you could link to one or more questions where you think this answer would be both on topic and at a more suitable level. Please bear in mind that "simple" is not a well-defined term and has differing subjective interpretations. Regardless, it would be invalid to conflate it with "mathematically unsophisticated," as suggested by your comments. $\endgroup$
    – whuber
    Jan 3, 2020 at 14:57
  • 7
    $\begingroup$ Xi'an's answer is not useless to me. I learned something. $\endgroup$
    – littleO
    Jan 3, 2020 at 17:24
3
$\begingroup$

I like using the notion of a "kernel" of a distribution. This is where you only leave in the parts that depend on the parameter. A few simple examples.

Normal kernel $$p(\mu|a,b) = K^{-1} \times \exp(a\mu^2 +b\mu)$$ Where $K$ is the "normalising constant" $K=\int \exp(a\mu^2 +b\mu)d\mu=\sqrt{\frac{\pi}{-a}}\exp(-\frac{b^2}{4a})$ The connection with standard mean/variance parameters is $E(\mu|a,b)=-\frac{b}{2a}$ and $Var(\mu|a,b)=-\frac{1}{2a}$

Beta kernel $$p(\theta|a,b)=K^{-1}\times \theta^a (1-\theta)^b$$ Where $K=\int \theta^a (1-\theta)^b d\theta = Beta(a+1,b+1)$

When we look at the likelihood function, we can do the same thing, and express it in "kernel form". For example with iid data

$$p(D|\mu)=\prod_{i=1}^n p(x_i|\mu)=Q\times f(\mu)$$

For some constant $Q$ and some function $f(\mu)$. If we can recognise this function as a kernel, then we can create a conjugate prior for that likelihood. If we take the normal likelihood with unit variance, the above looks like $$p(D|\mu) =\prod_{i=1}^n p(x_i|\mu) =\prod_{i=1}^n \frac{1}{\sqrt{2\pi}}\exp(-\frac{(x_i-\mu)^2}{2}) =\left[\prod_{i=1}^n \frac{1}{\sqrt{2\pi}}\right]\times \prod_{i=1}^n \exp(-\frac{(x_i-\mu)^2}{2}) =(2\pi)^{-\frac{n}{2}}\times\exp(-\sum_{i=1}^n\frac{(x_i-\mu)^2}{2}) =(2\pi)^{-\frac{n}{2}}\times\exp(-\sum_{i=1}^n\frac{x_i^2-2x_i\mu+\mu^2}{2}) =(2\pi)^{-\frac{n}{2}}\times\exp(-\sum_{i=1}^n\frac{x_i^2}{2})\times\exp(\mu\sum_{i=1}^n x_i-\mu^2\frac{n}{2}) =Q\times \exp(a\mu^2 +b\mu)$$

where $a=-\frac{n}{2}$ and $b=\sum_{i=1}^n x_i$ and $Q=(2\pi)^{-\frac{n}{2}}\times\exp(-\sum_{i=1}^n\frac{x_i^2}{2})$

This likelihood function has the same kernel as the normal distribution for $\mu$, so a conjugate prior for this likelihood is also the normal distribution. $$p(\mu|a_0,b_0)=K_0^{-1}\exp(a_0\mu^2 +b_0\mu)$$ The posterior is then $$p(\mu|D,a_0,b_0)\propto K_0^{-1}\exp(a_0\mu^2 +b_0\mu)\times Q\times \exp(a\mu^2 +b\mu)=K_0^{-1}\times Q\times \exp([a+a_0]\mu^2 +[b+b_0]\mu)\propto\exp([a+a_0]\mu^2 +[b+b_0]\mu)$$ Showing that the posterior is also a normal distribution, with updated parameters from the prior using the information in the data.

In some sense a conjugate prior acts similarly to adding "pseudo data" to the data observed, and then estimating the parameters.

$\endgroup$
1
  • 1
    $\begingroup$ (+1) I appreciate the pseudo-data intuition! $\endgroup$
    – Xi'an
    Jan 2, 2020 at 13:19
2
$\begingroup$

For a given distribution family $D_{lik}$ of the likelihood (e.g. Bernoulli),

if the prior is of the same distribution family $D_{pri}$ as the posterior (e.g. Beta),

then $D_{pri}$ and $D_{lik}$ are conjugate distribution families and the prior is called a conjugate prior for the likelihood function.

Note: $\underbrace{p(\theta|x)}_{\text{posterior}} \sim \underbrace{p(x|\theta)}_{\text{likelihood}} \cdot \underbrace{p(\theta)}_{\text{prior}}$

$\endgroup$
2
  • 1
    $\begingroup$ How does this explain what a conjugate prior is? $\endgroup$
    – LBogaardt
    Jan 2, 2020 at 11:05
  • $\begingroup$ ok I'll edit that. $\endgroup$
    – Tomas G.
    Jan 3, 2020 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.