# How to show that normal distribution is a second order approximation to any distribution around the mode?

How can I show that normal distribution is a second order approximation to any distribution around the mode?

• The Laplace approximation comes to mind : en.wikipedia.org/wiki/Laplace%27s_method Dec 10 '20 at 15:49
• Depending on what you mean by "second order approximation," the statement is false. If you are referring to density functions, then counterexamples include any distribution that does not have a differentiable density at its mode. If you are referring to CDFs, then counterexamples include all distributions of random variables that have nonzero probability of equaling their mode.
– whuber
Dec 10 '20 at 18:54

Consider an arbitrary probability distribution $$p(\theta)$$. We want to approximate the distribution around the mode $$\hat{\theta} = argmax \log p(\theta)$$. We perform a second order Taylor expansion around $$\hat{\theta}$$ in log space and we obtain $$\log p(\theta) = \log p(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta})^T \underbrace{\left( \nabla \nabla^T \log p(\hat{\theta})\right)}_{=: \psi} (\theta-\hat{\theta}) + O(\theta^3)$$ Where $$\psi$$ is the hessian matrix. Note that the first order term disappears as we are at a maximum and hence the first derivative is zero there. This is very similar to a Gaussian pdf in normal space, in fact $$p(\theta) \approx p(\hat{\theta}) \cdot \exp(- \frac{1}{2}(\theta - \hat{\theta})^T(-\psi) (\theta - \hat{\theta}))$$
Then we can define the Laplace approximation $$q$$ of $$p$$ as $$q(\theta) = \mathcal{N}(\theta, \hat{\theta}, -\psi^{-1})$$ It should be clear that this is proportional to the second order Taylor expansion. May also note that the hessian at the mode is symmetric and negative definite, hence $$-\psi$$ is symmetric and positive definite.
• Should the LHS of your equation be $\log p(\theta)$ rather than $\log p(\delta)$? Dec 10 '20 at 16:30
• Yeah introducing the $\delta$ was unnecessary, changed it. Dec 10 '20 at 17:52