I am reading a classic paper by Barron and Risannen, entitled:

The Minimum Description Length Principle in Coding and Modeling

In this paper they state the following idea: enter image description here

On this point I am interested in how they derive that the determinant of the Fisher Information Matrix (FIM) evaluates to the multiplication of the inverse of the histogram probability values.

It's probably a simple idea but I seem to have a conceptual block ....

Thus far, I am convinced that the FIM should be a diagonal matrix, which means that the determinant is just the multiplication of the diagonal elements. However, I can't imagine how the inverse probabilities arise.


1 Answer 1


We know that the $(i,j)^{th}$ element of the Fisher information matrix is defined as

$$ [\mathcal{I}(p)]_{i,j} = -E \left[ \frac{\partial^2}{\partial p_i \partial p_j} \log f(y|p,m) \right] $$

$f(y|p,m) = mp_{i(y)}$, so that $\log f(y|p,m) = \log m + \log p_{i(y)}$.

Derivating with respect to $p_i$ yields

\begin{equation} \frac{\partial}{\partial p_i } \log p_{i(y)}=\left\{ \begin{array}{@{}ll@{}} 0, & \text{if}\ i(y)\neq i \\ \frac{1}{p_{i(y)}}, & \text{otherwise} \end{array}\right. \end{equation}


\begin{equation} \frac{\partial^2}{\partial p_i \partial p_i } \log p_{i(y)}=\left\{ \begin{array}{@{}ll@{}} 0, & \text{if}\ i(y)\neq i \\ -\frac{1}{p_{i(y)}^2}, & \text{otherwise} \end{array}\right. \end{equation}

Also, $\frac{\partial^2}{\partial p_i \partial p_j } \log p_{i(y)} = 0$ for any $j \neq i$.

We finally have to compute the statistical expectation of this quantity:

$$ [\mathcal{I}(p)]_{i,j} = \int_{i(y) = i}dy \frac{1}{p_{i(y)}^2} f(y|p,m) = \int_{i(y) = i}dy \frac{1}{p_{i}^2} m p_{i} = \frac{1}{p_i} $$ The FIM is thus a diagonal matrix which $i^{th}$ entry is $p_i^{-1}$ ; and, as you mentioned, its determinant will be their product.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.