# Confusion about the Hessian approximation

There are two popular forms of the neural network Hessian approximation in the literature: $$H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right) \left(\frac{\partial y}{\partial w_i}\right)^T \quad H \simeq \frac{1}{n} \sum_i \left(\frac{\partial L}{\partial w_i}\right) \left(\frac{\partial L}{\partial w_i}\right)^T$$ First one approximates the Hessian via outer product of the Jacobians, and the second one is called Fisher approximation.

$$y$$ is the output of NN and $$L$$ is the value of loss function.

The origin of the first is simple to undertstand (see Outer product approximation of the Hessian). The term $$(y_n - t_n)$$ is small close to the optimum and one neglects it.

The second one originates from the Information theory and general expression: $$\mathbb{E}_{x, y \sim p(x, y)} \left[\nabla_w \log(y | x, w) (\nabla_w \log(y | x, w))^T\right]$$ And with the substitution $$\log(y | x, w) = -L(y, f(x, w))$$ one derives the second formula (replacing the true Fisher by empirical approximation): $$\mathbb{E}_{x, y \sim p(x, y)} \left[\nabla_w L (\nabla_w L)^T\right] = \frac{1}{n} \sum_i \left(\frac{\partial L}{\partial w_i}\right) \left(\frac{\partial L}{\partial w_i}\right)^T$$ Here loss function is evaluated on a single sample. Intuitively the statement is the follows. $$Hessian \simeq Variance \ of \ the \ gradient$$ However, it is not clear, when this approximation is valid.

Seems like there is some confusion in the literature. Recent review https://arxiv.org/abs/2102.00554 says, that the authors of the OBS method (Hassibi and Stork) use the Fisher approximation. But according to the original paper (https://authors.library.caltech.edu/54981/1/Optimal%20Brain%20Surgeon%20and%20general%20network%20pruning.pdf) first form of approximation is used.

If the gradient in the Fisher form was calculated on the whole batch, these two approaches would produce entirely different results, since $$\left(\frac{\partial L}{\partial w_i}\right) = \left(\frac{\partial L}{\partial y}\right) \left(\frac{\partial y}{\partial w_i}\right) = (y_n - t_n) \left(\frac{\partial y}{\partial w_i}\right)$$ Actually, this Hessian would vanish for well optimized model, and it is crucial, that the batch is small, I suppose.

Under what conditions, these two formulas are likely to produce close results?