1
$\begingroup$

Suppose I wish to compute the (asymptotic) Kullback-Leibler Divergence (KLD) between the exact Bayesian posterior $$q_{n}(\theta|x_{1:n}) \propto \pi(\theta)\prod_{i=1}^n p(x_i|\theta)$$ and the prior $\pi$ upon which it is based in closed form, irrespective of the likelihood model $p(x_i|\theta)$ used to arrive at $q_n$. Suppose also that we know that $q_n$ itself satisfies a Bernstein-von-Mises Theorem, i.e. is asymptotically normal with the same distribution as the Maximum Likelihood Estimator (MLE). In other words, I want to have an asymptotically exact approximation to $$ \mathbb{E}_{q_n}\left[ \sum_{i=1}^n\log(p(x_i|\theta))\right]$$ Is it feasible to apply a Laplace transformation for each term, i.e. to approximate the above as $$\left[ \pi(\hat{\theta}_n)e^{\sum_{j=1}^n\log(p(x_j|\hat{\theta}_n)) } \left( \dfrac{2\pi}{n} \right)^{d/2} |\mathcal{I}(\hat{\theta}_n, x)|^{-1/2} \left(\sum_{i=1}^n \log(p(x_i|\hat{\theta}_n)) %+ \log(\pi(\hat{\theta}_n)) \right) \right]\cdot \left( 1 + O\left(n^{-1}\right)\right) $$ where I have dropped/ignored the normalizing constant of $q_n$, where $\hat{\theta}$ is the MLE and $\mathcal{I}(\hat{\theta}_n, x)$ the Fisher information matrix given as usual by $$\mathcal{I}(\hat{\theta}_n, x) = -\dfrac{1}{n}\dfrac{\partial^2 \sum_{j=1}^n\log(p(x_j|\theta))}{\partial\theta\partial\theta^T}\Big|_{\theta = \hat{\theta}_n}$$ I cannot find anything wrong with the steps I take here, but if I use this approximation in a proof I arrive at a contradiction. What am I overlooking here? Any help in pointing me to the problem is very much appreciated!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.