15
$\begingroup$

I have been trying to understand the idea of conjugate priors in Bayesian statistics for a while but I simply don't get it. Can anyone explain the idea in the simplest possible terms, perhaps using the "Gaussian prior" as an example?

$\endgroup$
15
$\begingroup$

A prior for a parameter will almost always have some specific functional form (written in terms of the density, generally). Let's say we restrict ourselves to one particular family of distributions, in which case choosing our prior reduces to choosing the parameters of that family.

For example, consider a normal model $Y_i \stackrel{_\text{iid}}{\sim} N(\mu,\sigma^2)$. For simplicity, let's also take $\sigma^2$ as known. This part of the model - the model for the data - determines the likelihood function.

To complete our Bayesian model, here we need a prior for $\mu$.

As mentioned above, commonly we might specify some distributional family for our prior for $\mu$ and then we only have to choose the parameters of that distribution (for example, often prior information may be fairly vague - like roughly where we want the probability to concentrate - rather than of very specific functional form, and we may have enough freedom to model what we want by choosing the parameters - say to match a prior mean and variance).

If it turns out that the posterior for $\mu$ is from the same family as the prior, then that prior is said to be "conjugate".

(What makes it turn out to be conjugate is the way it combines with the likelihood)

So in this case, let's take a Gaussian prior for $\mu$ (say $\mu\sim N(\theta,\tau^2)$). If we do that, we see that the posterior for $\mu$ is also Gaussian. Consequently, the Gaussian prior was a conjugate prior for our model above.

That's all there is to it really -- if the posterior is from the same family as the prior, it's a conjugate prior.

In simple cases you can identify a conjugate prior by inspection of the likelihood. For example, consider a binomial likelihood; dropping the constants, it looks like a beta density in $p$; and because of the way powers of $p$ and $(1-p)$ combine, it will multiply by a beta prior to also give a product of powers of $p$ and $(1-p)$ ... so we can see immediately from the likelihood that the beta will be a conjugate prior for $p$ in the binomial likelihood.

In the Gaussian case it's easiest to see that it will happen by considering the log-densities and the log-likelihood; the log-likelihood will be quadratic in $\mu$ and the sum of two quadratics is quadratic, so a quadratic log-prior + quadratic log-likelihood gives a quadratic posterior (each of the coefficients of the highest order term will of course be negative).

$\endgroup$
5
$\begingroup$

If your model belongs to an exponential family, that is, if the density of the distribution is of the form $$f(x|\theta)=h(x)\exp\{T(\theta)\cdot S(x)-\psi(\theta)\}\qquad x\in\mathcal{X}\quad\theta\in\Theta$$ with respect to a given dominating measure (Lebesgue, counting, &tc.), where $t\cdot s$ denotes a scalar product over $\mathbb{R}^d$ and $$T:\mathcal{X}\longrightarrow \mathbb{R}^d\qquad S:\Theta\longrightarrow \mathbb{R}^d$$ are measurable functions, the conjugate priors on $\theta$ are defined by densities of the form $$\pi(\theta|\xi,\lambda)=C(\xi,\lambda)\exp\{T(\theta)\cdot \xi-\lambda\psi(\theta)\}$$ [with respect to an arbitrary dominating measure $\text{d}\nu$ on $\Theta$] with $$C(\xi,\lambda)^{-1}=\int_\Theta \exp\{T(\theta)\cdot \xi-\lambda\psi(\theta)\} \text{d}\nu<\infty$$ and $\lambda\in\Lambda\subset\mathbb{R}_+$, $\xi\in\Xi\subset \lambda T(\mathcal{X})$

Outside this setting, there is no non-trivial family of distributions with a fixed support that allows for conjugate priors. This is a consequence of the Darmois-Pitman-Koopman lemma.

$\endgroup$
  • 8
    $\begingroup$ "in simplest possible terms?" Perhaps an explanation that does not assume prior knowledge of measures would be more useful to the OP. $\endgroup$ – user44764 Oct 13 '15 at 20:21
  • 1
    $\begingroup$ alas, I am afraid conjugate priors are meaningless without a measure background (even though this is the best kept secret in the Universe). $\endgroup$ – Xi'an Oct 13 '15 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.