How can I show that normal distribution is a second order approximation to any distribution around the mode?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ The Laplace approximation comes to mind : en.wikipedia.org/wiki/Laplace%27s_method $\endgroup$ – Camille Gontier Dec 10 '20 at 15:49
-
1$\begingroup$ Depending on what you mean by "second order approximation," the statement is false. If you are referring to density functions, then counterexamples include any distribution that does not have a differentiable density at its mode. If you are referring to CDFs, then counterexamples include all distributions of random variables that have nonzero probability of equaling their mode. $\endgroup$ – whuber♦ Dec 10 '20 at 18:54
Add a comment
|
$\begingroup$
$\endgroup$
3
Consider an arbitrary probability distribution $p(\theta)$. We want to approximate the distribution around the mode $\hat{\theta} = argmax \log p(\theta)$. We perform a second order Taylor expansion around $\hat{\theta}$ in log space and we obtain $$ \log p(\theta) = \log p(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta})^T \underbrace{\left( \nabla \nabla^T \log p(\hat{\theta})\right)}_{=: \psi} (\theta-\hat{\theta}) + O(\theta^3) $$ Where $\psi$ is the hessian matrix. Note that the first order term disappears as we are at a maximum and hence the first derivative is zero there. This is very similar to a Gaussian pdf in normal space, in fact $$ p(\theta) \approx p(\hat{\theta}) \cdot \exp(- \frac{1}{2}(\theta - \hat{\theta})^T(-\psi) (\theta - \hat{\theta}))$$
Then we can define the Laplace approximation $q$ of $p$ as $$ q(\theta) = \mathcal{N}(\theta, \hat{\theta}, -\psi^{-1})$$ It should be clear that this is proportional to the second order Taylor expansion. May also note that the hessian at the mode is symmetric and negative definite, hence $-\psi$ is symmetric and positive definite.
-
$\begingroup$ Should the LHS of your equation be $\log p(\theta)$ rather than $\log p(\delta)$? $\endgroup$ – fblundun Dec 10 '20 at 16:30
-
$\begingroup$ Yeah introducing the $\delta$ was unnecessary, changed it. $\endgroup$ – Manuel Glöckler Dec 10 '20 at 17:52
-
$\begingroup$ Second order Taylor expansions do not necessarily exist. $\endgroup$ – whuber♦ Dec 10 '20 at 18:55
