Skip to main content
Cross Validated

Return to Question

added 127 characters in body
Source Link
Hohyun Kim
Hohyun Kim
  • 1
  • 1

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite

$$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2 \log p_\theta( x)] \\ &= E_{x \sim p_\theta}[\nabla_\theta\log p_\theta( x)(\nabla_\theta\log p_\theta( x))^T] \end{aligned} $$

If this is the case, my reasoning goes, any neural network that outputs negative log likelihood loss should also have positive semi-definite Hessians with regard to its parameters, because Hessians (i.e. observed Fisher information), when averaged, should be similar to the Fisher Information Matrix.

But it is contradiction to the fact that neural networks are non-convex. 

Please help me resolve this contradiction. Thanks!

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite

$$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2 \log p_\theta( x)] \\ &= E_{x \sim p_\theta}[\nabla_\theta\log p_\theta( x)(\nabla_\theta\log p_\theta( x))^T] \end{aligned} $$

If this is the case, my reasoning goes, any neural network that outputs negative log likelihood loss should also have positive semi-definite Hessians with regard to its parameters. But it is contradiction to the fact that neural networks are non-convex. Please help me resolve this contradiction. Thanks!

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite

$$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2 \log p_\theta( x)] \\ &= E_{x \sim p_\theta}[\nabla_\theta\log p_\theta( x)(\nabla_\theta\log p_\theta( x))^T] \end{aligned} $$

If this is the case, my reasoning goes, any neural network that outputs negative log likelihood loss should also have positive semi-definite Hessians with regard to its parameters, because Hessians (i.e. observed Fisher information), when averaged, should be similar to the Fisher Information Matrix.

But it is contradiction to the fact that neural networks are non-convex. 

Please help me resolve this contradiction. Thanks!

Source Link
Hohyun Kim
Hohyun Kim
  • 1
  • 1

Is Hessian of neural nets with NLL loss positive semi-definite?

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite

$$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}[-\nabla_\theta^2 \log p_\theta( x)] \\ &= E_{x \sim p_\theta}[\nabla_\theta\log p_\theta( x)(\nabla_\theta\log p_\theta( x))^T] \end{aligned} $$

If this is the case, my reasoning goes, any neural network that outputs negative log likelihood loss should also have positive semi-definite Hessians with regard to its parameters. But it is contradiction to the fact that neural networks are non-convex. Please help me resolve this contradiction. Thanks!