# computing KL divergence: M projections for arbitrary distributions

Background

I have a generative model for a process that can be described as follows:

$$y = t(x, w) + e$$

where $x$ and $y$ observations of a set of random variables which are related by a non-linear transformation function $t$, parameterised by the unknown parameters to be estimated given by $w$. $e$ is the normally distributed error term with 0 mean and a diagonal covariance matrix given by $\sigma^{-1} I$ where $I$ is the identity matrix and $\sigma$ is the global noise precision.

So, assuming independence along each pixel of the image, I have the following likelihood term as a product over the individual pixels $i$:

$$P(y|x, w, \sigma) = \prod_{i} (\frac{\sigma}{2\pi})^{\frac{1}{2}} \exp^{-\frac{1}{2}e_i \sigma e_i}$$

Each $w_i$ parameter is a 3-dimensional parameter (for each spatial dimension) to be estimated.

KL-Divergence and M-Projections:

I am using Expectation propagation (EP) to estimate the posterior distribution and EP has a M-projection step which projects a distribution onto a simpler approximating distribution which in my case is a multivariate normal distribution over my parameters $w$. The way EP works is by exploiting the factored form of the likelihood term i.e. it starts with an approximating distribution $Q$ (multivariate normal over $w$) and then replaces each ith factor by the exact term and projecting this distribution on the current estimate of the Q.

For example, in this problem I replace one ith term from my likelihood expression to form the distribution to project as: $P(y|x_i, w_i, \sigma) q_{j \neq i}(w_j)$ where the first term is the exact factor and $q_{j \neq i}(w_j)$ is the approximated distribution with the influence of the ith term removed. Let us call this distribution $U$

According to the literature, I need to find the moments $E_U[w]$. So, I need to match the first and second order moments i.e. compute these expectations for my parameters using the distribution $U$. However, I am completely lost as to how to do this. Can someone give me some suggestion on how I should proceed?