I have a setup where the joint posterior is written as:

$$ P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda) $$

Now $\phi$ and $\lambda$ are modelled as Gamma distributions and the likelihood and prior on $w$ is normally distributed. So, taking advantage of the conjugate prior property, the posterior on $\lambda$ and $\phi$ should be Gamma distributed.

So, my strategy is that for a given approximation for $w$, I will update the posterior parameters for $\lambda$ and $\phi$. So, for example, if I want to update $\lambda$, I would do something like:

$$ P(\lambda) = \int \int P(w, \lambda, \phi \vert y) \:dw \: d\phi $$ So, I marginalise out $w$ and $\phi$, evaluate the integral and match the terms to get the desired Gamma distribution parameters.

Now, I want to do it iteratively, so for a given instance 't' I have $w=w_t$. So, I would want to update the parameters of $\lambda$ and $\phi$ by conditioning on $w=w_t$. I have two very basic questions about this:

1: In the integral, can I basically replace the terms $w$ with $w_t$ and treat them like constants wrt to the variable of interest? I am slightly weak on the calculus side. This should be the case but I wanted to check. So, $P(\lambda|w=w_t) \propto \int \int P(w_t, \lambda, \phi \vert y) \: d\phi$

2: Also, $P(\lambda \vert w=w_t) = \frac{P(\lambda, w)}{P(w=w_t)}$ which is equal to $\frac{\int P(w, \lambda, \phi) \: d\phi}{P(w=w_t)}$. However, since the posterior $P(\lambda)$ is Gamma distributed, I am guessing I can always compute the normalisation factor at the end and do not have to worry about the denominator. Is that correct? So, all I want to do at the moment, is update the parameters and compute the expectation of the variable $\lambda$ and I guess the scaling should not matter in this case.


1 Answer 1

  1. Yes, to condition on a specific $w_t$ you can just replace the terms and consider them constant.
  2. If you know (by conjugacy) that the posterior for a parameter has a specific distribution, you can safely ignore the constant terms when you derive an expression for the posterior distribution. You can then find the normalization term in the definition of the distribution.

I have a few comments though:

It is not clear from the question that you are in a conjugate setting. For example, I assume that $w$ is Normal (Gaussian) distributed with zero mean and precision (inverse variance) $\lambda$, but this is not clear in your notation. Also, what is the mean and precision of your Normal likelihood? You might consider providing the full specification of your model.

Usually, with a Normal likelihood with unknown mean and precision, the conjugate prior is a Normal-Gamma (a joint prior for the mean and precision). In that case, the posterior is also Normal-Gamma, and there is no need for any iterative updating to compute it.

Kevin Murphy has a nice note on conjugate Gaussian models which might be worth looking at.

  • $\begingroup$ Thanks for the answer. Yes, the prior over $w$ is distributed with zero mean and the variance is a linear function of $\lambda$ and the distribution over $\lambda$ is Gamma distributed. I got this from page 101 of Chris Bishop's book where he describes the Gaussian gamma distribution (eq. 2.154). The noise precision $\phi$ is modelled with Gamma. The reason for the iteration is because the likelihood is not a function of $w$ but of $t(w)$, where $t$ is a highly non-linear transformation. Hence, I cannot use the Gaussian machinery and need to approximate $w$ and update $\lambda$ and $\phi$. $\endgroup$
    – Luca
    Mar 11, 2014 at 15:52
  • $\begingroup$ If you comments on the above, please let me know as the discussion will really help me! I will accept your answer in a while (just to keep the thread visible). Thanks again! $\endgroup$
    – Luca
    Mar 11, 2014 at 15:53
  • $\begingroup$ Just to add to my comment...even though $w$ is a multivariate normal distribution, the $\lambda$ and $\phi$ are assumed to be globally identically distributed. So, that is why I can use a multivariate normal-Gamma distribution to model the prior distribution $p(w, \lambda)$. $\endgroup$
    – Luca
    Mar 11, 2014 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.