I am trying to use Expectation propagation (EP) for approximating a posterior distribution in the Gaussian family. In this case, it is done by finding the Gaussian distribution with the same first and second moments as the arbitrary posterior distribution.

Assuming that the approximating distribution to be a family of Gaussians parametrised in terms of its mean $\mu$ and covariance $\Sigma$, i.e. $q(x) = \mathcal{N}(x; \mu, \Sigma)$, for the arbitrary distribution which is of the form $p(x) \propto t(x) q(x)$, the matching of the mean and covariance depends on the tractability of the normalising constant $Z = \int t(x) q(x; \mu, \Sigma) \,dx$. Here, t(x) is an arbitrary factor. So, what is happening is that we refine our approximation by taking into account each of the individual factors that make up the posterior distribution (assuming the posterior distribution can be written as a product of factors).

Looking at some papers, the estimate for the mean can be given by:

$$ \langle x \rangle_{p(x)} = \mu + \Sigma \nabla_{\mu}log\left(Z(\mu, \Sigma)\right) $$

Now, my question is computing this expression $\nabla_{\mu}log\left(Z(\mu, \Sigma)\right)$. Here is what I did:

$$ Z(\mu, \Sigma) = \int t(x)q(x; \mu, \Sigma) dx $$

$$ \nabla_{\mu}log\left(Z(\mu, \Sigma)\right) = \frac{\nabla_{\mu}Z(\mu, \Sigma)}{Z(\mu, \Sigma)} $$

Considering the numerator and using the linearity of the gradient operator,

$$ \begin{align*} \nabla_{\mu}Z(\mu, \Sigma) &= \int \left[\nabla_{\mu} t(x) q(x; \mu, \Sigma)\right] \, dx \\ &= \int t(x) \, \Sigma^{-1}\,(x-\mu)\,q(x; \mu, \Sigma) dx \end{align*} $$

So, overall I get some expression as:

$$ \nabla_{\mu}log\left(Z(\mu, \Sigma)\right) = \frac{\int t(x) \, \Sigma^{-1}\,(x-\mu)\,q(x; \mu, \Sigma) dx}{\int t(x)q(x; \mu, \Sigma) dx} $$

I was wondering if there is a way to simplify this further. The way it is at the moment, it seems pretty intractable. Perhaps, I am going about it completely the wrong way. Any help or suggestion would be greatly appreciated.


You need to substitute the particular $t(x)$ that you are trying to approximate, get the formula for $Z$, then take derivatives. See the examples in A family of algorithms for approximate Bayesian inference and EP: A quick reference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.