What is the difference between 'Laplace approximation' and 'Modified harmonic mean'?

Question

this question is about Bayesian and computational statistics. I am learning them right now, I have two very common output from my software, one is Laplace approximation and the other is Modified harmonic mean. Both of them are used for approximation of log marginal likelihood, usually they have very close value. What are their differences? And could anyone provide me some background knowledge no these methods? Thanks!

Answer 1

I assume, you are refering to the estimation of marginal likelihood. Laplace approximation is used to approximate a marginal likelihood based on normal distribution. Resulting estimate is as follows: $$f(y)\approx (2\pi)^{d}|\widetilde{\Sigma}|^{1/2}f(y|\widetilde{\theta})f(\widetilde{\theta})$$ Where $\widetilde{\theta}$ is a posterior mode (can be estimated from MCMC output) and $\widetilde{\Sigma}$ is inverse of Hessian matrix based on log-likelihood and evaluated at posterior mode (this can be evaluated from MCMC output as well), $f(y|\theta)$ your likelihood and $f(\theta)$ is prior distribution. This approximation works well if your posterior distribution is similar to normal distribution. The second estimator, modified (generalized) harmonic mean estimator is derived by the following identity: $$\frac{1}{f(y)}=\int \frac{1}{f(y)}g(\theta)d \theta=\int \frac{1}{f(y|\theta)f(\theta)}f(\theta|y)g(\theta)d \theta=\int \frac{g(\theta)}{f(y|\theta)f(\theta)}f(\theta|y)d \theta=E_{f(\theta|y)}\left [ \frac{g(\theta)}{f(y|\theta)f(\theta)} \right ]$$ Here $g(\theta)$ is any "importance" density and has to be close to the posterior. So these two, Laplace and generalized harmonic mean, estimators are for marginal likelihood value estimation.
However, they are of different origins and behave differently: Laplace estimator is approximation-based estimator and will not work if your posterior is not bell-shaped distribution (multimodal or highly skewd). Otherwise it works quite well. Generlized harmonic mean estimator is based on exact identity and theoretically it should provided accurate estimates, however, as its close relative harmonic mean estimator, it is quite sensitive. Do not know more about its performance, but its relative harmonic mean estimator is usually very badly-behaved estimator. I do not use it in my work.
Hope I answered your question.