Is Hessian of neural nets with NLL loss positive semi-definite?

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite

\begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2 \log p_\theta( x)] \\ &= E_{x \sim p_\theta}[\nabla_\theta\log p_\theta( x)(\nabla_\theta\log p_\theta( x))^T] \end{aligned}

If this is the case, my reasoning goes, any neural network that outputs negative log likelihood loss should also have positive semi-definite Hessians with regard to its parameters, because Hessians (i.e. observed Fisher information), when averaged, should be similar to the Fisher Information Matrix.

But it is contradiction to the fact that neural networks are non-convex.

• H=F only near the global optimum (where the Hessian must be positive definite). Commented Jul 27, 2022 at 22:04
• @JohnMadden I wanted to know why that's the case, but you helped me figuring out the answer anyway. Thanks! Commented Jul 30, 2022 at 0:38

Hessian of the loss is defined over data distribution whereas Fisher information is defined over model's distribution:

\begin{aligned} H(\theta) &= E_{x \sim Data}[-\nabla_\theta^2\log p_\theta (x)] \\ &= E_{x \sim Data}[-\frac{\nabla_\theta^2 p_\theta}{p_\theta} + \nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T] \\ & \neq E_{x \sim Data}[\nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T] \end{aligned}

which is defferent from

\begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2\log p_\theta (x)] \\ &= E_{x \sim p_\theta}[-\frac{\nabla_\theta^2 p_\theta}{p_\theta} + \nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T] \\ &= E_{x \sim p_\theta}[\nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T]. \end{aligned}

Actually, many people seems to ignore this distinction, which led them to use "empirical Fisher". See [Kunstner et al., 2019], which clearly draws distinctions between various terms on Fisher information, and points out possible problems on using empirical Fisher inadvertantly.

Reference:

Kunstner et al. Limitations of the Empirical Fisher Approximation for Natural Gradient Descent. Neurips. 2019.