16
$\begingroup$

What are the definitions of semi-conjugate priors and of conditional conjugate priors? I found them in Gelman's Bayesian Data Analysis, but I couldn't find their definitions.

$\endgroup$

3 Answers 3

19
$\begingroup$

Using the definition in Bayesian Data Analysis (3rd ed), if $\mathcal{F}$ is a class of sampling distributions $p(y|\theta)$, and $\mathcal{P}$ is a class of prior distributions for $\theta$, then the class $\mathcal{P}$ is conjugate for $\mathcal{F}$ if

$$p(\theta|y)\in \mathcal{P} \mbox{ for all }p(\cdot|\theta)\in \mathcal{F} \mbox{ and }p(\cdot)\in \mathcal{P}.$$

If $\mathcal{F}$ is a class of sampling distributions $p(y|\theta,\phi)$, and $\mathcal{P}$ is a class of prior distributions for $\theta$ conditional on $\phi$, then the class $\mathcal{P}$ is conditional conjugate for $\mathcal{F}$ if

$$p(\theta|y,\phi)\in \mathcal{P} \mbox{ for all }p(\cdot|\theta,\phi)\in \mathcal{F} \mbox{ and }p(\cdot|\phi)\in \mathcal{P}.$$

Conditionally conjugate priors are convenient in constructing a Gibbs sampler since the full conditional will be a known family.

I searched an electronic version of Bayesian Data Analysis (3rd ed.) and could not find a reference to semi-conjugate prior. I'm guessing it is synonymous with conditionally conjugate, but if you provide a reference to its use in the book, I should be able to provide a definition.

$\endgroup$
7
  • $\begingroup$ +1. What's the URL for the 3rd edition of Bayesian Data Analysis? $\endgroup$ Mar 22, 2014 at 2:11
  • 1
    $\begingroup$ Thanks! Semi-conjugate appears here (2nd ed) books.google.com/…. By the way, how did you get the ebook for the 3rd ed? $\endgroup$
    – Tim
    Mar 22, 2014 at 3:15
  • 1
    $\begingroup$ I'm not sure why it says semiconjugate prior there since the prior is fully conjugate. This statement is removed in the 3rd edition. The ebook can be purchased here: crcpress.com/product/isbn/9781439840955. $\endgroup$
    – jaradniemi
    Mar 22, 2014 at 22:37
  • $\begingroup$ @jaradniemi: In the link I gave, on top of p84, it is pointed out that the semiconjugate prior is not a conjugate prior. $\endgroup$
    – Tim
    Mar 22, 2014 at 22:52
  • 2
    $\begingroup$ In $$p(\theta|y,\phi)\in \mathcal{P} \mbox{ for all }p(\cdot|\theta,\phi)\in \mathcal{F} \mbox{ and }p(\cdot|\phi)\in \mathcal{P}.$$ what do each of the $\cdot$ refer to and does each refer to the same thing? $\endgroup$
    – Muno
    Mar 15, 2018 at 16:35
8
$\begingroup$

I would like to use multivariate normal as an example.

Recall that the likelihood is given by

$$ P(y_1,y_2,...,y_n|\mu,\Sigma) = (2\pi)^{-\frac{ND}{2}}\det(\Sigma)^{-\frac{N}{2}}\exp(\frac{1}{2}\sum_{i=1}^N(x_i-\mu)^T\Sigma^{-1}(x_i-\mu)) $$

In order to find a prior to this likelihood, we may choose

$$ P(\mu,\Sigma)=\text{Normal}(\mu;\mu_0,\Lambda_0)\text{InverseWishart}(\Sigma;\nu_0,S_0) $$

I assure you NOT to worry about $\mu_0,\Lambda_0,\nu_0,S_0$ for now; they are simply parameters of the prior distribution.

What is important is, however, that this is not conjugate to the likelihood. To see why, I would like to quote a reference I found online.

note that $\mu$ and $\Sigma$ appear together in a non-factorized way in the likelihood; hence they will also be coupled together in the posterior

The reference is "Machine Learning: A Probabilistic Perspective" by Kevin P. Murphy. Here is the link. You may find the quote in Section 4.6 (Inferring the parameters of an MVN) at the top of page 135.

To continue the quote,

The above prior is sometimes called semi-conjugate or conditionally conjugate, since both conditionals, $p(\mu|\Sigma)$ and $p(\Sigma|\mu)$, are individually conjugate. To create a full conjugate prior, we need to use a prior where $\mu$ and $\Sigma$ are dependent on each other. We will use a joint distribution of the form

$$ p(\mu, \Sigma) = p(\Sigma)p(\mu|\Sigma) $$

The idea here is that the first prior distribution

$$ P(\mu,\Sigma)=\text{Normal}(\mu;\mu_0,\Lambda_0)\text{InverseWishart}(\Sigma;\nu_0,S_0) $$

assumes that $\mu$ and $\Sigma$ are separable (or independent in a sense). Nevertheless, we observe that in the likelihood function, $\mu$ and $\Sigma$ cannot be factorized out separately, which implies that they will not be separable in the posterior (Recall, $(\text{Posterior}) \sim (\text{Prior})(\text{Likelihood})$). This shows that the "un-separable" posterior and "separable" prior at the beginning are not conjugate. On the other hand, by rewriting

$$ p(\mu, \Sigma) = p(\Sigma)p(\mu|\Sigma) $$

such that $\mu$ and $\Sigma$ depend on each other (through $p(\mu|\Sigma)$), you will obtain a conjugate prior, which is named as semi-conjugate prior. This hopefully answers your question.

p.s.: Another really helpful reference I have used is "A First Course in Bayesian Statistical Methods" by Peter D. Hoff. Here is a link to the book. You may find relevant content in Section 7 starting from page 105, and he has a very good explanation (and intuition) about single-variate normal distribution in Section 5 starting from page 67, which will be reinforced again in Section 7 when he deals with MVN.

$\endgroup$
2
$\begingroup$

If $F$ is a class of sampling distributions $p(y|θ,ϕ)$, and $P$ is a class of prior distributions for $θ$, then the class $P$ is semiconjugate for $F$ if $p(θ|y,ϕ)∈P$ for all $p(⋅|θ,ϕ)∈F$ and $p(θ,ϕ)=p(θ)\times p(ϕ)$, where $p(θ)∈P$ and $p(ϕ)$ does not belong to class $P$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.