Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

this question is about Bayesian and computational statistics. I am learning them right now, I have two very common output from my software, one is Laplace approximation and the other is Modified harmonic mean. Both of them are used for approximation of log marginal likelihood, usually they have very close value. What are their differences? And could anyone provide me some background knowledge no these methods? Thanks!

share|improve this question

1 Answer 1

up vote 6 down vote accepted

I assume, you are refering to the estimation of marginal likelihood. Laplace approximation is used to approximate a marginal likelihood based on normal distribution. Resulting estimate is as follows:

$$f(y)\approx (2\pi)^{d}|\widetilde{\Sigma}|^{1/2}f(y|\widetilde{\theta})f(\widetilde{\theta})$$

Where $\widetilde{\theta}$ is a posterior mode (can be estimated from MCMC output) and $\widetilde{\Sigma}$ is inverse of Hessian matrix based on log-likelihood and evaluated at posterior mode (this can be evaluated from MCMC output as well), $f(y|\theta)$ your likelihood and $f(\theta)$ is prior distribution. This approximation works well if your posterior distribution is similar to normal distribution. The second estimator, modified (generalized) harmonic mean estimator is derived by the following identity: $$\frac{1}{f(y)}=\int \frac{1}{f(y)}g(\theta)d \theta=\int \frac{1}{f(y|\theta)f(\theta)}f(\theta|y)g(\theta)d \theta=\int \frac{g(\theta)}{f(y|\theta)f(\theta)}f(\theta|y)d \theta=E_{f(\theta|y)}\left [ \frac{g(\theta)}{f(y|\theta)f(\theta)} \right ]$$ Here $g(\theta)$ is any "importance" density and has to be close to the posterior. So these two, Laplace and generalized harmonic mean, estimators are for marginal likelihood value estimation.
However, they are of different origins and behave differently: Laplace estimator is approximation-based estimator and will not work if your posterior is not bell-shaped distribution (multimodal or highly skewd). Otherwise it works quite well. Generlized harmonic mean estimator is based on exact identity and theoretically it should provided accurate estimates, however, as its close relative harmonic mean estimator, it is quite sensitive. Do not know more about its performance, but its relative harmonic mean estimator is usually very badly-behaved estimator. I do not use it in my work.
Hope I answered your question.

share|improve this answer
1  
Thanks so much! Would you please recommend a textbook for Bayesian computation? –  Flying pig Dec 3 '12 at 22:30
1  
An anonymous editor contends that there is an error here:$$f(y)\approx n^{-d/2}(2\pi)^{d/2}|\widetilde{\Sigma}|^{1/2}f(y|\widetilde{\theta})f( \widetilde{ \theta})$$, citing this pdf –  gung Aug 23 at 21:18
1  
I'd suggest taking a look at some of Christian Robert's comments on harmonic mean estimators at his blog (Xi'an's Og) –  Glen_b Aug 23 at 22:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.