How to show that normal distribution is a second order approximation to any distribution around the mode? How can I show that normal distribution is a second order approximation to any distribution around the mode?
 A: Consider an arbitrary probability distribution $p(\theta)$. We want to approximate the distribution around the mode $\hat{\theta} = argmax \log p(\theta)$. We perform a second order Taylor expansion around $\hat{\theta}$ in log space and we obtain
$$ \log p(\theta) = \log p(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta})^T \underbrace{\left( \nabla \nabla^T \log p(\hat{\theta})\right)}_{=: \psi} (\theta-\hat{\theta}) + O(\theta^3) $$
Where $\psi$ is the hessian matrix. Note that the first order term disappears as we are at a maximum and hence the first derivative is zero there. This is very similar to a Gaussian pdf in normal space, in fact
$$ p(\theta) \approx p(\hat{\theta}) \cdot \exp(- \frac{1}{2}(\theta - \hat{\theta})^T(-\psi) (\theta - \hat{\theta}))$$
Then we can define the Laplace approximation $q$ of $p$ as
$$ q(\theta) = \mathcal{N}(\theta, \hat{\theta}, -\psi^{-1})$$
It should be clear that this is proportional to the second order Taylor expansion. May also note that the hessian at the mode is symmetric and negative definite, hence $-\psi$ is symmetric and positive definite.
