13
$\begingroup$

Do all conjugate priors have to come from the exponential family? If not, what other families are known to have/produce conjugate priors?

$\endgroup$
0

1 Answer 1

17
$\begingroup$

As explained for example in Section 3.3.3 of book "The Bayesian choice" by Christian Robert, there is indeed a narrow connection between exponential families and conjugate priors, but there are conjugate priors available for certain non-exponential families. He calls these "quasi-exponential", however, because they are families for which sufficient statistics of finite dimension not increasing with sample size do exist.

Here is an example for the uniform distribution, whose support depends on the parameter of the distribution and can therefore not be an exponential family (as is well-known):

Here, the Pareto distribution is a conjugate prior for the parameter $b$ of the uniform distribution on $[0,b]$.

The density of the Pareto distribution with parameters $c>0$ and $% \alpha >0$ is \begin{equation*} f(x)=\alpha c^{\alpha }x^{-\alpha -1} \end{equation*} for $x\geq c$ and $f(x)=0$ else.

The prior of the parameter $b$ of a uniform distribution on $[0,b]$ is a Pareto distribution with $c_{0}$ and $\alpha _{0}$, \begin{eqnarray*} \pi (b) &=& \left\{ \begin{array}{ll} \alpha _{0}c_{0}^{\alpha _{0}}b ^{-\alpha _{0}-1} & \quad \text{if } b \geq c_{0} \\ 0 & \quad \text{else.}% \end{array} \right. \\ &\propto &\left\{ \begin{array}{ll} b ^{-\alpha _{0}-1} & \quad \text{if }b \geq c_{0} \\ 0 & \quad \text{else.}% \end{array}% \right. \end{eqnarray*} The likelihood for the data $y_{1},\ldots ,y_{n}$, given $b$, is \begin{equation*} f\left( y |b\right) = \left\{ \begin{array}{ll} \prod_{i=1}^{n}\frac{1}{b }=b ^{-n} & \quad \text{if }0\leq y_{i}\leq b \text{ for all }i=1,\ldots ,n \\ 0 & \quad \text{else.}% \end{array}% \right. \end{equation*} The product of likelihood and prior is the non-normalized posterior \begin{eqnarray*} \pi \left( b |y\right) &\propto &\pi (b )f\left( y |b\right) \\ &=&\left\{ \begin{array}{ll} \alpha _{0}c_{0}^{\alpha _{0}}b ^{-\alpha _{0}-1}b ^{-n} & \quad \text{if }b \geq c_{0}\text{ and }0\leq y_{i}\leq b \text{ for all }i=1,\ldots ,n \\ 0 & \quad \text{else.}% \end{array}% \right. \\ &\propto &\left\{ \begin{array}{ll} b ^{-\alpha _{0}-n-1} & \quad \text{if }b \geq c_{0}\text{ and }% 0\leq y_{i}\leq b \text{ for all }i=1,\ldots ,n \\ 0 & \quad \text{else.}% \end{array}% \right. \\ &\propto &\left\{ \begin{array}{ll} b ^{-\alpha _{1}-1} & \quad \text{if }b \geq c_{1} \\ 0 & \quad \text{else.}% \end{array}% \right. \end{eqnarray*} with \begin{eqnarray*} \alpha _{1} &=&\alpha _{0}+n \\ c_{1} &=&\max\bigl(\max_{i}y_{i},c_0\bigr). \end{eqnarray*} Hence, the posterior is Pareto distributed.

$\endgroup$
8
  • 5
    $\begingroup$ (+1) So iff there's a sufficient statistic of constant dimension, there's a conjugate prior? $\endgroup$ Jan 27, 2016 at 10:48
  • 5
    $\begingroup$ Very interesting question - I do not know! My answer just provides an example that membership of an exponential family is not a necessary condition for the existence of a conjugate prior. I would be very interested in the answer, so please ask this as a separate question! $\endgroup$ Jan 27, 2016 at 10:53
  • 3
    $\begingroup$ I've a feeling it's got to be so for updating to work. I'll certainly ask a question if I can't find a book answer. $\endgroup$ Jan 27, 2016 at 11:11
  • 4
    $\begingroup$ @Scortchi: yes indeed, because if there exists a sufficient statistic of fixed dimension then we are in an exponential family, as established by the Pitman-Koopman-Darmois lemma. $\endgroup$
    – Xi'an
    Mar 12, 2016 at 16:38
  • 4
    $\begingroup$ Does this not omit the qualifier: "among all families whose support does not depend on the parameter", see also the above example? $\endgroup$ Mar 12, 2016 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.