Expectation Propagation and multivariate priors

Question

I have been struggling with this for weeks now. All EP examples that I have found on the net seem to deal with univariate priors and I am really at a loss as to how to make it work with a multivariate prior. The scenario is as follows: I am trying to estimate a bunch of parameters where the prior over these parameters form a multivariate Gaussian with off diagonal covariance terms. The likelihood can be written down as a product over individual likelihood for each of these parameters. So, the equation is as follows:

$$ P(w|y) = \frac{1}{Z}P(w)\prod_iP(y_i|w_i) $$

where $y$ is the observed data and $w$ are the parameters I am trying to estimate. So, when I want to perform inference on this using EP, what should be done with $P(w)$. $P(w)$ is a multivariate Gaussian with 0 mean and a dense covariance matrix. So, at some point I will have to multiply terms between $P(w)$ and $P(y_i|w_i)$ or some similar approximates of it (but the forms are very different). So effectively to compute the functional form of the posterior, I am multiplying a multivariate Gaussian ($P(w)$) with a univariate Gaussian function ($\prod_i{q(y_i|w_i)}$) where $q(y_i|w_i)$ are the approximating factors. How can this be achieved? Does $P(w)$ need to be broken down as well? I am not sure how to make this work.

Answer 1

Allow me to change $w$ to $W$ to differentiate it from the observed variables. Based off the information in the linked question from the comments you have

$$\begin{eqnarray*} P(W|x, y,\phi) &\propto& P(W,x,y,\phi)\\ &\propto& P(W,y | x, \phi)\\ &=& P(y| W,x,\phi)P(W | x, \phi)\\ &=& P(y | W,x,\phi)P(W) \end{eqnarray*}$$

So up to normalization the information you seek is $$P(W|x,y,\phi) \propto \left[\prod_{n=1}^N\frac{\phi}{2\pi}e^{-.5(y_n-t(x_n,W_n))\phi(y_n-t(x_n,W_n))}\right] \frac{1}{\sqrt{(2\pi)^N|\Sigma|}}e^{-.5(W-m)^T\Sigma^{-1}(W-m)}$$

The normalized result is $$P(W|x,y,\phi) = \left[\prod_{n=1}^N\frac{\phi}{2\pi}e^{-.5(y_n-t(x_n,W_n))\phi(y_n-t(x_n,W_n))}\right] \frac{1}{\sqrt{(2\pi)^N|\Sigma|}}e^{-.5(W-m)^T\Sigma^{-1}(W-m)}P(x)P(\phi)$$ If you would like I could comment about why you don't need to use EP in this problem.