I have a joint probability distribution as given in the figure:

enter image description here

In this figure, variables in circles are random variables and variables in squares are constants. So, I can write the joint distribution over the data $y$ and the model parameters as:

$$ P(y, w, \lambda, \phi) = P(y|w, \phi) \times P(w|\lambda) \times P(\lambda) \times P(\phi) $$

Now $P(w|\lambda)$ is modelled using a multivariate normal distribution with 0 mean and a covariance matrix scaled by $\lambda$. $P(\lambda)$ and $P(\phi)$ are modelled using gamma distributions. Also, the likelihood term $P(y|x, w, \phi)$ is a Gaussian likelihood given by:

$$ P(y|x, w, \phi) = (\frac{\phi}{2\pi})^{0.5} \exp^{-0.5 e \phi e} $$

The model noise is independent and identically distributed.

Now, I am interested in $P(w, \lambda, \phi|y)$ which is given by the joint distribution above normalised appropriately by $P(y)$

My question is regarding the use of expectation propagation to perform inference on this model? I have been trying to understand EP with little success.

Can someone help me understand what approximations I need to make to this model to use EP on it? The prior $P(w|\lambda)$ is modelled using a zero mean multivariate normal distribution. $P(\lambda)$ is modelled using a Gamma distribution and $P(\phi)$ is also a gamma distribution. So, to infer the posterior $P(w, \lambda, \phi|y)$, I am confused as to where to start. Should I start with the mean field like approach to assume independence among the posterior distributions of these parameters?

$$ P(w, \lambda, \phi) \approx q(w) \times q(\lambda) \times q(\phi) $$

I have been struggling with this for weeks now. Any suggestion/reference etc. would be really appreciated. Looking at Minka's lecture notes, I am having issues figuring out whether some factor graph representation for this as that is usually the structure he uses in his examples.


1 Answer 1


The factor graph representation is already implied by your first equation, which writes the joint distribution as a product of factors. Thus there are four factors.

  • $\begingroup$ Thanks! If the likelihood can be written down as a product over image pixels because of IID assumption of the noise model, is it better to break it down into many factors? So, I infer a Gaussian at every image pixel independently or is it usually better to try and estimate them all in one factor? Thanks again! I thoroughly enjoy your lectures (I struggle with it though). $\endgroup$
    – Luca
    Jan 15, 2014 at 15:23
  • $\begingroup$ Generally you would give each pixel its own factor. The only reason to group them is if you could get some sort of computational savings, which only happens for very particular kinds of likelihoods. $\endgroup$
    – Tom Minka
    Jan 15, 2014 at 15:27
  • 1
    $\begingroup$ Thanks Tom! You are a star. You should visit more often :-) $\endgroup$
    – Luca
    Jan 15, 2014 at 15:29
  • $\begingroup$ What does one do when the prior is a multivariate normal distribution with a covariance structure and the likelihood can be written as a product of univariate normals. How do you multiply to get the leave one out approximation then? $\endgroup$
    – Luca
    Jan 16, 2014 at 17:41
  • $\begingroup$ Hi Tom, I wonder if you come to this site often or not. But I have one fundamental question about the EP algorithm as described in your papers. When we remove the old estimate of a factor, we are removing it usually from a high dimensional posterior distribution whereas an individual factor itself is usually just specified by a mean and standard deviation. So how does this division operator i.e. $\frac{q(\theta)}{t_i}$ work as $q(\theta)$ is typically high dimensional but the factor $t_i$ is not. $\endgroup$
    – Luca
    Feb 12, 2014 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.