This is continuing the discussion about the posterior probability densities and conjugate priors.

So, my posterior distribution over the parameters is defined as:

$$ P(w, \phi, \lambda | y) \propto P(\phi) P(w|\lambda) P(\lambda) \prod_{i}P(y|w, \phi, \lambda) $$

Now, $\phi$ is the parameter that models the noise precision. Noise is assumed to be global zero mean and IID distributed. So, the mean is known and we have an unknown precision and the conjugate prior is given by a Gamma distribution. So, $P(\phi) = Ga(\phi; a, b)$.

Now, the distribution over $w$ where both the mean and precision are unknown, we can write $P(w, \lambda)$ as $P(w|\lambda)P(\lambda)$. As described in Bishop's book, to achieve conjugacy, $P(w|\lambda)$ is modelled as a zero mean MVN whose precision is a function of $\lambda$ given by $(\lambda \Lambda)^{-1}$. Also, $P(\lambda)$ is modelled using a gamma distribution i.e. $P(\lambda) = Ga(\lambda; c, d)$.

So, now my questions are about what the posterior distribution.

1: Due to the use of such conjugate priors, is the posterior over all the parameters $P(w, \phi, \lambda | y)$ easier to $\textit{sample}$ from or is there a closed form expression for it which obviates the need for sampling all together and I can estimate the sufficient statistics analytically?

2: Can I say something that this joint posterior will have some particular form or can I only say something about the marginal distributions i.e. $P(w)$ will be normally distributed posteriorly, $P(\phi)$ will be gamma distributed etc. What about the sufficient statistics of these marginals? Can they be computed analytically? What about dependence/coupling of the parameters in the posterior distribution? I am guessing since we have coupling of $\lambda$ and $w$ in our model, it should be coupled in the posterior distribution as well. However, to perform any sort of approximate inference on such a system, do we $\textit{have}$ to using mean field approximations and assume independence between the parameters in the posterior distribution?


1 Answer 1


Sorry for answering my own question but I hope it will be helpful for someone else.

I missed a very fundamental point. The prior and the likelihood are not defined over the same parameters. In the example, I gave the prior is given by $P(w)$ and the likelihood is related to $w$ in a more complex and non-linear way. So, the likelihood is normal with respect to residuals (equivalent to minimising sum of square difference). So, they are normal but over completely different parameters and hence the posterior distribution will not be normal. At least this how I understand it after giving it much thought.

  • $\begingroup$ +1 Thanks for answering. We won't be able to spot problems like that if we can't see the form of the likelihood. $\endgroup$
    – Glen_b
    Feb 19, 2014 at 8:21
  • $\begingroup$ Oh absolutely. It was me not understanding the model. $\endgroup$
    – Luca
    Feb 19, 2014 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.