2
$\begingroup$

How can I show that normal distribution is a second order approximation to any distribution around the mode?

$\endgroup$
2
  • 1
    $\begingroup$ The Laplace approximation comes to mind : en.wikipedia.org/wiki/Laplace%27s_method $\endgroup$ – Camille Gontier Dec 10 '20 at 15:49
  • 1
    $\begingroup$ Depending on what you mean by "second order approximation," the statement is false. If you are referring to density functions, then counterexamples include any distribution that does not have a differentiable density at its mode. If you are referring to CDFs, then counterexamples include all distributions of random variables that have nonzero probability of equaling their mode. $\endgroup$ – whuber Dec 10 '20 at 18:54
2
$\begingroup$

Consider an arbitrary probability distribution $p(\theta)$. We want to approximate the distribution around the mode $\hat{\theta} = argmax \log p(\theta)$. We perform a second order Taylor expansion around $\hat{\theta}$ in log space and we obtain $$ \log p(\theta) = \log p(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta})^T \underbrace{\left( \nabla \nabla^T \log p(\hat{\theta})\right)}_{=: \psi} (\theta-\hat{\theta}) + O(\theta^3) $$ Where $\psi$ is the hessian matrix. Note that the first order term disappears as we are at a maximum and hence the first derivative is zero there. This is very similar to a Gaussian pdf in normal space, in fact $$ p(\theta) \approx p(\hat{\theta}) \cdot \exp(- \frac{1}{2}(\theta - \hat{\theta})^T(-\psi) (\theta - \hat{\theta}))$$

Then we can define the Laplace approximation $q$ of $p$ as $$ q(\theta) = \mathcal{N}(\theta, \hat{\theta}, -\psi^{-1})$$ It should be clear that this is proportional to the second order Taylor expansion. May also note that the hessian at the mode is symmetric and negative definite, hence $-\psi$ is symmetric and positive definite.

$\endgroup$
3
  • $\begingroup$ Should the LHS of your equation be $\log p(\theta)$ rather than $\log p(\delta)$? $\endgroup$ – fblundun Dec 10 '20 at 16:30
  • $\begingroup$ Yeah introducing the $\delta$ was unnecessary, changed it. $\endgroup$ – Manuel Glöckler Dec 10 '20 at 17:52
  • $\begingroup$ Second order Taylor expansions do not necessarily exist. $\endgroup$ – whuber Dec 10 '20 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.