2
$\begingroup$

**

“Any meaning to the concept of ‘Self Mutual Information?”

**

A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very good explanation and summary of the concepts of Mutual Information, KL Divergence and their relationships to Entropy. It also contained many informative references as well as providing useful Python code supporting their explanations. But portions of the results from one set of code are puzzling to me. The code calculated the mutual information between the features for a dataset describing cancer. It created an “MI” (Mutual Information) matrix by interating across all the features, thus covering all possible combinations of the features in the data set (see screenshot below for MI matrix resulting from the code).

screenshot of results from code that produces an MI Matrix with both redundant MI from reciprocal pairs of features (mirrored across the diagonal of the matrix like "Age - Insulin" and "Insulin - Age") as well as the self pairs like "Age - Age" down the diagonal

I understand that the reciprocal pairs of features that are mirrored across the diagonal of the matrix, like “Insulin - Resistin” and “Resistin - Insulin”, are redundant but, do the MI of pairs of “self-feature pairs” like “Insulin - Insulin” or “Resistin - Resistin” provide any useful information?

I look forward to any feedback or insights.

$\endgroup$

1 Answer 1

1
$\begingroup$

What you called Self Mutual Information is actually Entropy. Indeed entropy can be also called Self Information (wouldn't use "mutual" anymore): https://en.wikipedia.org/wiki/Information_content

$$ \mbox{MI}(X,X) = H(X) - H(X|X) = H(X) - \sum_x \sum_x p(x,x) \log{p(x|x)} = H(X) - \sum_x \sum_x p(x,x) \log{1} = H(X) $$

Thus, on the diagonal of that table you will find the entropy of each variable:

  • High entropy: values are uniformly distributed
  • 0 entropy: one single constant value
$\endgroup$
1
  • 2
    $\begingroup$ Perfect answer and reference - thanks! $\endgroup$ Jan 2, 2020 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.