Suppose I wish to compute the (asymptotic) Kullback-Leibler Divergence (KLD) between the exact Bayesian posterior $$q_{n}(\theta|x_{1:n}) \propto \pi(\theta)\prod_{i=1}^n p(x_i|\theta)$$ and the prior $\pi$ upon which it is based in closed form, irrespective of the likelihood model $p(x_i|\theta)$ used to arrive at $q_n$. Suppose also that we know that $q_n$ itself satisfies a Bernstein-von-Mises Theorem, i.e. is asymptotically normal with the same distribution as the Maximum Likelihood Estimator (MLE). In other words, I want to have an asymptotically exact approximation to $$ \mathbb{E}_{q_n}\left[ \sum_{i=1}^n\log(p(x_i|\theta))\right]$$ Is it feasible to apply a Laplace transformation for each term, i.e. to approximate the above as $$\left[ \pi(\hat{\theta}_n)e^{\sum_{j=1}^n\log(p(x_j|\hat{\theta}_n)) } \left( \dfrac{2\pi}{n} \right)^{d/2} |\mathcal{I}(\hat{\theta}_n, x)|^{-1/2} \left(\sum_{i=1}^n \log(p(x_i|\hat{\theta}_n)) %+ \log(\pi(\hat{\theta}_n)) \right) \right]\cdot \left( 1 + O\left(n^{-1}\right)\right) $$ where I have dropped/ignored the normalizing constant of $q_n$, where $\hat{\theta}$ is the MLE and $\mathcal{I}(\hat{\theta}_n, x)$ the Fisher information matrix given as usual by $$\mathcal{I}(\hat{\theta}_n, x) = -\dfrac{1}{n}\dfrac{\partial^2 \sum_{j=1}^n\log(p(x_j|\theta))}{\partial\theta\partial\theta^T}\Big|_{\theta = \hat{\theta}_n}$$ I cannot find anything wrong with the steps I take here, but if I use this approximation in a proof I arrive at a contradiction. What am I overlooking here? Any help in pointing me to the problem is very much appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.