I have a dataset $\mathbf{D} = \{ (\tau_i, \Gamma_i) : 1 \le i \le n \}$ of observations $\tau_i = X_i + \epsilon_i$ from a $p$-dimensional Gaussian $X_i \sim \mathcal{N}(\mu, \Sigma)$ contaminated by additive noise $\epsilon_i \sim \mathcal{N}(0, \Gamma_i)$ where the $\Gamma_i$ are diagonal and observed. This gives a likelihood function of $$ \begin{aligned} L(\mu,\Sigma) &= \prod_i \int_{\mathbb{R}^p} \mathcal{N}(x_i | \tau_i, \Gamma_i)\, \mathcal{N}(x_i | \mu, \Sigma) \,dx_i \\ &= \prod_i \mathcal{N}(\tau_i | \mu, \Sigma+\Gamma_i). \end{aligned} $$

Normally I would use a Normal-inverse-Wishart prior to get a Multivariate Student's t predictive distribution, but it's not clear how to do that with this likelihood.

Is it at least possible to get a "sensible" approximation to the predictive distribution of $X$? Is there there some other prior I should consider?


1 Answer 1


This problem is inherently difficult for Bayesian inference. Changing the prior is not going to help. You will need to resort to approximate inference techniques such as Monte Carlo, variational methods, or normal approximations.

  • $\begingroup$ Hi, Tom! Actually, this problem is motivated by an application of the Expectation Propagation algorithm, but I'm stuck on constructing a decent approximate distribution on $X$. The $(\tau_i, \Gamma_i)$ are computed using EP to approximate the true likelihood function, and I'd like to then incorporate these into updating the beliefs on $X$. I see how to get a Gaussian predictive distribution on $X$ if I choose the prior to be $\mu \sim \mathcal{N}(0, \Sigma_0)$ if $\Sigma_0$ is "known", but not how to get a reasonable posterior with both $\mu$ and $\Sigma$ unknown. Thanks. $\endgroup$
    – Pine Tree
    Mar 10, 2014 at 18:27
  • $\begingroup$ Like I said, you will have to do some additional approximate inference. Since you are already using EP, then you might try using it to approximate the posterior on $\mu$ and $\Sigma$ given this likelihood. But I think mean-field would be simpler and quicker. $\endgroup$
    – Tom Minka
    Mar 11, 2014 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.