# Correlation is to covariance, what mutual information is to --?

The information theory equivalent of the correlation matrix is the mutual information matrix, which has individual entropies along its diagonal, and mutual information estimates in the off-diagonals. It captures non-linear interactions between variable pairs unlike its classical counterpart.

Since covariance is a simple transformation of correlation, $$Cov(X, Y) = Cor(X, Y) \times \sqrt{Var(X)Var(Y)}$$, isn't there also an equivalent of the covariance matrix in information theory? If not, what is there that's closest? or what does the mutual information lack to similarly describe what covariance does?

On the other hand correlation and "normalized" mutual information -- wikipedia suggests $$\frac{I(X;Y)}{\sqrt{H(X)H(Y)}}$$, among a number of other options -- are both bounded (correlation between -1 and 1, and normalized mutual information between 0 and 1), and are also unitless.
• if mutual information (MI) is to covariance what normalized normalized information (NMI) is to correlation because NMI is bounded between 0 and 1, then the MI matrix is to the variance-covariance matrix ($\boldsymbol{\Sigma}$) what the NMI matrix is to the correlation matrix. what can be said about variance in the diagonals of $\boldsymbol{\Sigma}$ always being bounded at 0, whereas off-diagonals aren't? the diagonals of the MI matrix are entropies. Also, does the MI matrix mimic how covariance takes on negative values if a pair-wise correlation (NMI) is negative? Aug 12 '20 at 3:16