Is Hessian of neural nets with NLL loss positive semi-definite?

Question

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite

$$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2 \log p_\theta( x)] \\ &= E_{x \sim p_\theta}[\nabla_\theta\log p_\theta( x)(\nabla_\theta\log p_\theta( x))^T] \end{aligned} $$

If this is the case, my reasoning goes, any neural network that outputs negative log likelihood loss should also have positive semi-definite Hessians with regard to its parameters, because Hessians (i.e. observed Fisher information), when averaged, should be similar to the Fisher Information Matrix.

But it is contradiction to the fact that neural networks are non-convex.

Please help me resolve this contradiction. Thanks!

Answer 1

Hessian of the loss is defined over data distribution whereas Fisher information is defined over model's distribution:

$$ \begin{aligned} H(\theta) &= E_{x \sim Data}[-\nabla_\theta^2\log p_\theta (x)] \\ &= E_{x \sim Data}[-\frac{\nabla_\theta^2 p_\theta}{p_\theta} + \nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T] \\ & \neq E_{x \sim Data}[\nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T] \end{aligned} $$

which is defferent from

$$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2\log p_\theta (x)] \\ &= E_{x \sim p_\theta}[-\frac{\nabla_\theta^2 p_\theta}{p_\theta} + \nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T] \\ &= E_{x \sim p_\theta}[\nabla_\theta \log p_\theta(x) \nabla_\theta \log p_\theta(x)^T]. \end{aligned} $$

Actually, many people seems to ignore this distinction, which led them to use "empirical Fisher". See [Kunstner et al., 2019], which clearly draws distinctions between various terms on Fisher information, and points out possible problems on using empirical Fisher inadvertantly.

Reference:

Kunstner et al. Limitations of the Empirical Fisher Approximation for Natural Gradient Descent. Neurips. 2019.