I have a PGM as described by the attached diagram.

Graphical model

$y$ is observed and I want to infer the joint posterior distribution as given by $P(w,\lambda, \phi |y)$. Distribution on $w$ is modelled using a multivariate 0 mean with a covariance structure. $\lambda$ needs to be inferred from the data and the prior on it is modelled using a gamma distribution with scale and shape parameters $s_0$ and $C_0$. The noise variance $\phi$ also needs to be estimated and its prior is also modelled using a Gamma distribution with parameters $a_0$ and $b_0$

I have been looking at using Expectation Propagation to perform this approximate inference on this model. As I understand, EP works by minimising the KL divergence between the true posterior and the approximated distributions.

So, my questions are:

1: Can I use EP in such a case?

2: Can someone help me understand what objective function I need to design in this model? As far as I can tell, the EP will approximate the try posterior by independent distributions from the exponential family. So,

$P(w,\lambda, \phi) \approx q(w) q(\lambda) q(\phi)$

And now do we minimise the KL divergence between true p and q's? I am really having trouble understanding this setup and would be really grateful if someone can help me understand where and how to start!

Any help/pointer/comments will be appreciated.

Thanks, Luca


1 Answer 1


Yes, you can use EP for this. Start by writing the model as a factor graph. For each edge of the factor graph, there will be two messages: one from the variable to the factor and another from the factor to the variable. EP provides a formula for each message in terms of the surrounding messages. This gives a system of equations that you iterate until convergence. Once the messages have converged, you compute the marginal distributions from them. See the paper Divergence measures and message passing for details. There is a free software library called Infer.NET that implements EP message updates for many different factor types. You provide it with a factor graph and it generates code with the message updates.

  • $\begingroup$ Thanks so much for this! I guess the answer cannot come from a better source! Also, can the different approximating distributions for the variables be from different distributions (gamma, multivariate normal etc.) as long as they are from the exponential family? $\endgroup$
    – Luca
    Commented Jan 15, 2014 at 15:12
  • $\begingroup$ Yes, you can use different exponential families to approximate the marginal for different variables. You only need all messages into a variable to have the same type, so that they can be multiplied together to get the marginal. $\endgroup$
    – Tom Minka
    Commented Jan 15, 2014 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.