How (or can) you formulate the Fisher information matrix in terms of a loss function, specifically cross-entropy loss?

I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning: $$\mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}{\partial m}\right) \left(\frac{\partial \mathcal{L}(x,y;1)}{\partial m} \right)^T$$

$$\mathcal{L}(x, y; 1)$$ here refers to the loss function. However, the usual formulation of the Fisher information matrix, stolen from Wikipedia, is:

$$\bigl[\mathcal{I}(\theta)\bigr]_{i, j} = \operatorname{E}\left[\left. \left(\frac{\partial}{\partial\theta_i} \log f(X;\theta)\right) \left(\frac{\partial}{\partial\theta_j} \log f(X;\theta)\right) \,\, \right| \,\,\theta\right].$$ Where $$f(X;\theta)$$ is the parametrized function for the model. I'm not sure how these formulations are equivalent. While it seems to be presented as fact regardless of the specific loss function in the paper, for what it's worth the loss used is cross-entropy. I've tried deriving one from the other, but I'm not really able to.

• If you associate $\theta$ with $m$ and $\log f$ with $\mathcal{L}$, you're basically there, it seems to me. Oct 22, 2023 at 20:24
• The issue is that $\mathcal{L}$ isn't $\log f$, with cross-entropy loss it is $-E_{f(X)}[\log f(X;\theta)]$ Oct 23, 2023 at 11:22

I think the answer to your question comes from the definition of $$L(x,y;\mathbb{1})$$. To me, this looks like the loss over a single example for which only one term of the cross-entropy remains, i.e.,
$$L(x,y;\mathbb{1})=log P(y|x;\mathbb{1}),$$