I have been struggling with this for weeks now. All EP examples that I have found on the net seem to deal with univariate priors and I am really at a loss as to how to make it work with a multivariate prior. The scenario is as follows: I am trying to estimate a bunch of parameters where the prior over these parameters form a multivariate Gaussian with off diagonal covariance terms. The likelihood can be written down as a product over individual likelihood for each of these parameters. So, the equation is as follows:

$$ P(w|y) = \frac{1}{Z}P(w)\prod_iP(y_i|w_i) $$

where $y$ is the observed data and $w$ are the parameters I am trying to estimate. So, when I want to perform inference on this using EP, what should be done with $P(w)$. $P(w)$ is a multivariate Gaussian with 0 mean and a dense covariance matrix. So, at some point I will have to multiply terms between $P(w)$ and $P(y_i|w_i)$ or some similar approximates of it (but the forms are very different). So effectively to compute the functional form of the posterior, I am multiplying a multivariate Gaussian ($P(w)$) with a univariate Gaussian function ($\prod_i{q(y_i|w_i)}$) where $q(y_i|w_i)$ are the approximating factors. How can this be achieved? Does $P(w)$ need to be broken down as well? I am not sure how to make this work.


Allow me to change $w$ to $W$ to differentiate it from the observed variables. Based off the information in the linked question from the comments you have

$$\begin{eqnarray*} P(W|x, y,\phi) &\propto& P(W,x,y,\phi)\\ &\propto& P(W,y | x, \phi)\\ &=& P(y| W,x,\phi)P(W | x, \phi)\\ &=& P(y | W,x,\phi)P(W) \end{eqnarray*}$$

So up to normalization the information you seek is $$P(W|x,y,\phi) \propto \left[\prod_{n=1}^N\frac{\phi}{2\pi}e^{-.5(y_n-t(x_n,W_n))\phi(y_n-t(x_n,W_n))}\right] \frac{1}{\sqrt{(2\pi)^N|\Sigma|}}e^{-.5(W-m)^T\Sigma^{-1}(W-m)}$$

The normalized result is $$P(W|x,y,\phi) = \left[\prod_{n=1}^N\frac{\phi}{2\pi}e^{-.5(y_n-t(x_n,W_n))\phi(y_n-t(x_n,W_n))}\right] \frac{1}{\sqrt{(2\pi)^N|\Sigma|}}e^{-.5(W-m)^T\Sigma^{-1}(W-m)}P(x)P(\phi)$$ If you would like I could comment about why you don't need to use EP in this problem.

  • $\begingroup$ The reason I am confused (and this could be something simple that I do not grasp) is that as I understand in EP, we approximate the posterior distribution by iteratively refining our approximation for each factor. So we will have this local messages where at some point we will need to multiply these messages (AFAI can tell) and this has a factor with a multivariate normal prior and some univariate likelihood terms coming in and I am not sure how these things combine. A detailed question: link $\endgroup$ – Luca Feb 7 '14 at 23:07
  • $\begingroup$ With respect to the linked question, are the parameters $m$ and $\Sigma$ in $P(w)$ known? $\endgroup$ – SomeEE Feb 8 '14 at 0:28
  • $\begingroup$ This is the parameters of the prior distribution. So, the prior is a 0 mean MVN ($m$=0) with a known covariance structure $\Sigma$. $\endgroup$ – Luca Feb 8 '14 at 10:48
  • $\begingroup$ Ok - that is what I thought. See the updated answer. $\endgroup$ – SomeEE Feb 8 '14 at 17:40
  • $\begingroup$ Thanks! Yes, please comment on why EP is not needed :-) Is there some closed form solution? $\endgroup$ – Luca Feb 9 '14 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.