I'm assuming this is pretty commonly discussed but haven't found it yet.

For example, if you start with a binary (0-1 binomial), conjugate prior is beta distribution. Beta distribution has some un-named conjugate prior. You can take the conjugate prior of that.

Is there anything interesting coming of this? Anything useful? Looking for links and references mostly.


No, there is no end to this process, and you always end up with a distribution having one additional parameter.

Copied from my answer here:

Let's say that you have a distribution $F$ in the exponential family with density \begin{align} \newcommand{\mbx}{\mathbf x} \newcommand{\btheta}{\boldsymbol{\theta}} f(\mbx \mid \btheta) &= \exp\bigl(\eta(\btheta) \cdot T(\mbx) - g(\btheta) + h(\mbx)\bigr) \end{align}

Given independent realizations $\{x_1, x_2, \dotsc, x_n\}$ of $F$ (with unknown parameter $\theta$), then the distribution over $\theta$, $F'$, is the conjugate prior of $F$. The density of $F'$ is \begin{align} f(\btheta \mid \boldsymbol\phi) = L(\btheta \mid \mbx_1, \dotsc, \mbx_n) &= f(\mbx_1, \dotsc, \mbx_n \mid \btheta) \\\\ &\propto \prod_i f(\mbx_i\mid \btheta) \\\\ &= \textstyle\prod_i\exp\Bigl(\eta(\btheta) \cdot \textstyle T\left(\mbx_i\right) - g(\btheta) + h(\mbx_i)\Bigr) \\\\ &\propto \textstyle\prod_i\exp\Bigl(\eta(\btheta) \cdot \textstyle T\left(\mbx_i\right) - g(\btheta)\Bigr) \\\\ &= \textstyle\exp\Bigl(\eta(\btheta) \cdot \bigl(\textstyle\sum_iT\left(\mbx_i\right)\bigr) - ng(\btheta)\Bigr) \\\\ &= \exp\bigl(\eta'(\boldsymbol \phi) \cdot T'(\btheta)\bigr) \end{align} where \begin{align} \eta'(\boldsymbol\phi) &= \begin{bmatrix} \sum_iT_1(\mbx_i) \\\\ \vdots \\\\ \sum_iT_k(\mbx_i) \\\\ \sum_i1 \end{bmatrix} & T'(\btheta) &= \begin{bmatrix} \eta_1(\btheta) \\\\ \vdots \\\\ \eta_k(\btheta) \\\\ -g(\btheta) \end{bmatrix}. \end{align} Thus, $F'$ is also in the exponential family ($T'$ replaced $\eta$ and $\eta'$ replaced $T$ since this distribution is over $\theta$ the parameter of the distribution over $x$.)

Interestingly, $\boldsymbol\phi$ has exactly one more parameter than $\btheta$ except in the rare case where natural parameter $\phi_{k+1}$ is redundant, but such a distribution would be very weird (it would mean that the number of observations $\mbx$, that is, $n$, tells you nothing about $\btheta$.)

So, to answer your question, with each conjugate prior you get exactly one more hyperparameter.

| cite | improve this answer | |
  • 1
    $\begingroup$ (+1) Indeed, the next conjugate remains within an exponential family and the extra parameter that was a scale term gets another conjugate that is also within an exponential family. $\endgroup$ – Xi'an Aug 12 at 20:00
  • $\begingroup$ Interesting ... but isn't one of those parameters kind of the weight of the prior and you can set it to one or something fixed? If you do that or something like this are there any results on convergen of the sequence? $\endgroup$ – mathtick Aug 12 at 20:50
  • 1
    $\begingroup$ @mathtick See my edit. $\endgroup$ – Neil G Aug 12 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.