2
$\begingroup$

I've found in a paper here. Observed Fisher information estimated in a way that does not convince me at all. They estimated it as the covariance matrix of the scores, but to me the formula used is wrong:

$$\hat I(\theta)=\sum_{i=1}^n (\mathbf S_i\mathbf S_i^t)-\frac1n\left(\sum_{i=1}^n \mathbf S_i\right)\left(\sum_{i=1}^n \mathbf S_i^t\right)$$

I think that $1/n$ should be before the first term, and the second term should be simply the mean of the scores.

Can you confirm that this formula is wrong?

$\endgroup$
2
  • 1
    $\begingroup$ Here is a MathJax tutorial for typesetting math. Consider using MathJax whenever possible. $\endgroup$ Sep 1 '21 at 12:52
  • $\begingroup$ Thanks a lot, I didn't know how to do that. $\endgroup$ Sep 1 '21 at 13:00
2
$\begingroup$

The formula looks a bit unusual but note that \begin{align} \frac1n\sum_{i=1}^n(S_i-\bar S)(S_i-\bar S)^T &=\frac1n\sum_{i=1}^n (S_iS_i^T-2\bar S S_i^T+\bar S \bar S^T) \\&=\frac1n\sum_{i=1}^n S_iS_i^T-2\bar S\frac1n\sum_{i=1}^n S_i^T+\frac1n\bar S\bar S^T \\&=\frac1n\sum_{i=1}^n S_iS_i^T-\frac1{n^2}\sum_{i=1}^nS_i\sum_{i=1}^nS_i^T. \end{align} This can be seen as a kind of estimator of $\operatorname{Var}S_i$. If the observations (and the contributions $S_i$ to the score vector) are independent, $n$ times this is a kind of estimator of the expected (and I suppose the observed) Fisher information $$ I(\theta)=\operatorname{Var}S=\operatorname{Var}(\sum_{i=1}^n S_i)=n\operatorname{Var}S_i. $$ But if the likelihood is available, the observed Fisher information is directly observable so it is a bit strange that the authors try to only estimate it in this way.

Note also that the last term in their expression is not needed as $S=\sum S_i=0$ at the MLE of $\theta$.

This estimator (without the last term) is known as the empirical Fisher information in the machine learning literature. Kunstner et. al. 2019 argue "that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects."

$\endgroup$
1
  • $\begingroup$ Thanks a lot for your answer, it clarified my doubt. About your concern on how they have calculated the observed Fisher information, if you are interested, you can take a look at my other question here: link. Basically, the problem is that completed data log-likelihood is corrected, due to ascertainment bias, at E-Step of EM algorithm and it raises some problems. $\endgroup$ Sep 1 '21 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.