Can the off-diagonal elements of Fisher information matrix be negative?

Question

The concept of Fisher information is new to me and as I understand the diagonal elements of the Fisher information matrix (FIM) are proportional to mean square error (to be precise the inverse of FIM). What is the interpretation of the off-diagonal elements of FIM and can these be negative?

Answer 1

TL;DR: The information matrix can be diagonalized, in which case its diagonal elements are positive, whereas non-diagonal elements are zero.

A meaningful information matrix must be positive-definite, but there is no requirements that non-diagonal elements must be positive. That such elements are present indicates that the chosen variables are correlated, and one should exercise care when interpreting diagonal elements as mean square error. This interpretation is correct, if we diagonalize the information matrix, in which case the diagonal elements are positive, while non-diagonal elements are zero.

Answer 2

A counter-example: For a Beta $\mathcal B(\alpha,\beta)$ distribution, the Fisher information matrix on $(\alpha,\beta)$ is $$I(\alpha,\beta) = \left( \begin{matrix} \psi'(\alpha) -\psi'(\alpha+\beta) & -\psi'(\alpha+\beta)\\ -\psi'(\alpha+\beta) & \psi'(\beta) -\psi'(\alpha+\beta) \end{matrix}\right)$$ where $\psi(\cdot)$ is the digamma function and $\psi'(\cdot)>0$.

Answer 3

There is a link between Fisher information matrices and covariance matrices, see Demonstration and Interpretation between a Fisher matrix and its dual space which is covariance matrix for details. So asymptotically Fisher information can be interpreted as an inverse covariance matrix, that is, a precision matrix. For derails see How to interpret an inverse covariance or precision matrix? and especially Why does inversion of a covariance matrix yield partial correlations between random variables?.

Partial correlations can certainly be negative.