11
$\begingroup$

I'm trying to compute the marginal likelihood for a statistical model by Monte Carlo methods:

$$f(x) = \int f(x\mid\theta) \pi(\theta)\, d\theta$$

The likelihood is well behaved - smooth, log-concave - but high-dimensional. I've tried importance sampling, but the results are wonky and depend highly on the proposal I'm using. I briefly considered doing Hamiltonian Monte Carlo to compute posterior samples assuming a uniform prior over $\theta$ and taking the harmonic mean, until I saw this. Lesson learned, the harmonic mean can have infinite variance. Is there an alternative MCMC estimator that is nearly as simple, but has a well-behaved variance?

Improve this question
$\endgroup$
3
  • $\begingroup$ You could also consider basic monte carlo sampling from the prior. $f(x)=E_{\pi(\theta)}(f(x|\theta))$ $\endgroup$
    – probabilityislogic
    May 28, 2013 at 11:06
  • 1
    $\begingroup$ That's one possible solution. In this case, remember that improper priors are no longer allowed, and priors with a very spread support will probably make the Monte Carlo approximation difficult. $\endgroup$
    – Zen
    May 28, 2013 at 18:33
  • 1
    $\begingroup$ A complete book on the issue is Chen, Shao and Ibrahim (2001). You can also search for keywords like nested sampling, bridge sampling, defensive sampling, particle filters, Savage-Dickey. $\endgroup$
    – Xi'an
    Sep 10, 2016 at 9:13

2 Answers 2

Reset to default
8
$\begingroup$

How about annealed importance sampling? It has much lower variance than regular importance sampling. I've seen it called the "gold standard", and it's not much harder to implement than "normal" importance sampling. It's slower in the sense that you have to make a bunch of MCMC moves for each sample, but each sample tends to be very high-quality so you don't need as many of them before your estimates settle down.

The other major alternative is sequential importance sampling. My sense is that it's also fairly straightforward to implement, but it requires some familiarity with sequential Monte Carlo (AKA particle filtering), which I lack.

Good luck!

Edited to add: It looks like the Radford Neal blog post you linked to also recommends Annealed Importance Sampling. Let us know if it works well for you.

Improve this answer
$\endgroup$
2
$\begingroup$

This might help on sheding some light on marginal distribution calculation. Also, I would recommend to use a method through power posteriors introduced by Friel and Pettitt. This approach seems quite promissing, although it has some limitations. Or you could you Laplace approximation of posterior distribution by normal distribution: if histogram from MCMC looks symmetric and normal-like, than this could be quite good approximation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.