I would like to know why some paper uses Normalized Mutual Information and not standard Mutual Information to measure correlation between features? what is the difference between these two measures?


2 Answers 2

  • Mutual Information I(X,Y) yelds values from $0$ (no mutual information - variables X and Y are independent) to $+\infty$. The higher the I(X,Y), the more information is shared between X and Y. However, high values of mutual information might be unintuitive and hard to interpret due to its unbounded range of values $I(X,Y)\in [0...\infty)$.
  • Normalized Mutual Information measures try to bring the possible values to bounded range $I(X,Y)\in [0...m]$. Specifically, case of $m=1$ is useful due to ease of comparison with commonly used correlation coefficients.

Nice discussion of relations between Mutual Information and Pearson Correlation Coefficient can be found in Materials and Methods section of "Generalized Correlation for Biomolecular Dynamics" paper by Lange and Grubmuller[1]. They also introducte of generalized correlation coefficient that maps values of I(X,Y) onto [0,1] interval, which can be seen as another approach Normalised Mutual Information.

[1] O. F. Lange, H. Grubmüller, Proteins 2006, 62, 1053–1061.

  • 6
    $\begingroup$ Be wary of each author's definition of normalized mutual information. Depending on the field (or even the lab, group or country) this may be normalized using the joint entropy, the sum of marginal entropies or some other related quantity. In my experience working with hydrostatistics, the choice of normalization doesn't make an enormous difference unless the types of data vectors used have wildly varying behaviors. $\endgroup$ Commented Jul 9, 2015 at 23:28
  • $\begingroup$ Can you add an equation that describes NMI. Since there are few ones. Also maybe it could be good to compare few. $\endgroup$
    – Michael D
    Commented Aug 26, 2021 at 13:54

Unlike correlation, mutual information is not bounded always less then 1. Ie it is the number of bits of information shared between two variables and thus depends on the total information content of each of the variables.

Various measures of normalized Mutual Information are attempts to make it more like correlation by bounding it (ie 1 is good, 0 is bad).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.