# “Any meaning to the concept of ‘Self Mutual Information?”

**

## “Any meaning to the concept of ‘Self Mutual Information?”

**

A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very good explanation and summary of the concepts of Mutual Information, KL Divergence and their relationships to Entropy. It also contained many informative references as well as providing useful Python code supporting their explanations. But portions of the results from one set of code are puzzling to me. The code calculated the mutual information between the features for a dataset describing cancer. It created an “MI” (Mutual Information) matrix by interating across all the features, thus covering all possible combinations of the features in the data set (see screenshot below for MI matrix resulting from the code).

I understand that the reciprocal pairs of features that are mirrored across the diagonal of the matrix, like “Insulin - Resistin” and “Resistin - Insulin”, are redundant but, do the MI of pairs of “self-feature pairs” like “Insulin - Insulin” or “Resistin - Resistin” provide any useful information?

I look forward to any feedback or insights.

## 1 Answer

What you called Self Mutual Information is actually Entropy. Indeed entropy can be also called Self Information (wouldn't use "mutual" anymore): https://en.wikipedia.org/wiki/Information_content

$$\mbox{MI}(X,X) = H(X) - H(X|X) = H(X) - \sum_x \sum_x p(x,x) \log{p(x|x)} = H(X) - \sum_x \sum_x p(x,x) \log{1} = H(X)$$

Thus, on the diagonal of that table you will find the entropy of each variable:

• High entropy: values are uniformly distributed
• 0 entropy: one single constant value
• Perfect answer and reference - thanks! Jan 2, 2020 at 21:27