0
$\begingroup$

I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning: $$ \mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}{\partial m}\right) \left(\frac{\partial \mathcal{L}(x,y;1)}{\partial m} \right)^T $$

$\mathcal{L}(x, y; 1)$ here refers to the loss function. However, the usual formulation of the Fisher information matrix, stolen from Wikipedia, is:

$$ \bigl[\mathcal{I}(\theta)\bigr]_{i, j} = \operatorname{E}\left[\left. \left(\frac{\partial}{\partial\theta_i} \log f(X;\theta)\right) \left(\frac{\partial}{\partial\theta_j} \log f(X;\theta)\right) \,\, \right| \,\,\theta\right]. $$ Where $f(X;\theta)$ is the parametrized function for the model. I'm not sure how these formulations are equivalent. While it seems to be presented as fact regardless of the specific loss function in the paper, for what it's worth the loss used is cross-entropy. I've tried deriving one from the other, but I'm not really able to.

$\endgroup$
2
  • $\begingroup$ If you associate $\theta$ with $m$ and $\log f$ with $\mathcal{L}$, you're basically there, it seems to me. $\endgroup$
    – jbowman
    Oct 22, 2023 at 20:24
  • $\begingroup$ The issue is that $\mathcal{L}$ isn't $\log f$, with cross-entropy loss it is $-E_{f(X)}[\log f(X;\theta)]$ $\endgroup$
    – premed
    Oct 23, 2023 at 11:22

1 Answer 1

0
$\begingroup$

The definition of the paper you refer to is

The notation in the paper is

I think the answer to your question comes from the definition of $L(x,y;\mathbb{1})$. To me, this looks like the loss over a single example for which only one term of the cross-entropy remains, i.e.,

$$L(x,y;\mathbb{1})=log P(y|x;\mathbb{1}),$$

With this you get the equivalence between two definitions of the FIM.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.