There are two popular forms of the neural network Hessian approximation in the literature: $$ H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right) \left(\frac{\partial y}{\partial w_i}\right)^T \quad H \simeq \frac{1}{n} \sum_i \left(\frac{\partial L}{\partial w_i}\right) \left(\frac{\partial L}{\partial w_i}\right)^T $$ First one approximates the Hessian via outer product of the Jacobians, and the second one is called Fisher approximation.
$y$ is the output of NN and $L$ is the value of loss function.
The origin of the first is simple to undertstand (see Outer product approximation of the Hessian). The term $(y_n - t_n)$ is small close to the optimum and one neglects it.
The second one originates from the Information theory and general expression: $$ \mathbb{E}_{x, y \sim p(x, y)} \left[\nabla_w \log(y | x, w) (\nabla_w \log(y | x, w))^T\right] $$ And with the substitution $\log(y | x, w) = -L(y, f(x, w))$ one derives the second formula (replacing the true Fisher by empirical approximation): $$ \mathbb{E}_{x, y \sim p(x, y)} \left[\nabla_w L (\nabla_w L)^T\right] = \frac{1}{n} \sum_i \left(\frac{\partial L}{\partial w_i}\right) \left(\frac{\partial L}{\partial w_i}\right)^T $$ Here loss function is evaluated on a single sample. Intuitively the statement is the follows. $$ Hessian \simeq Variance \ of \ the \ gradient $$ However, it is not clear, when this approximation is valid.
Seems like there is some confusion in the literature. Recent review https://arxiv.org/abs/2102.00554 says, that the authors of the OBS method (Hassibi and Stork) use the Fisher approximation. But according to the original paper (https://authors.library.caltech.edu/54981/1/Optimal%20Brain%20Surgeon%20and%20general%20network%20pruning.pdf) first form of approximation is used.
If the gradient in the Fisher form was calculated on the whole batch, these two approaches would produce entirely different results, since $$ \left(\frac{\partial L}{\partial w_i}\right) = \left(\frac{\partial L}{\partial y}\right) \left(\frac{\partial y}{\partial w_i}\right) = (y_n - t_n) \left(\frac{\partial y}{\partial w_i}\right) $$ Actually, this Hessian would vanish for well optimized model, and it is crucial, that the batch is small, I suppose.
Under what conditions, these two formulas are likely to produce close results?
