3
$\begingroup$

The information theory equivalent of the correlation matrix is the mutual information matrix, which has individual entropies along its diagonal, and mutual information estimates in the off-diagonals. It captures non-linear interactions between variable pairs unlike its classical counterpart.

Since covariance is a simple transformation of correlation, $ Cov(X, Y) = Cor(X, Y) \times \sqrt{Var(X)Var(Y)}$, isn't there also an equivalent of the covariance matrix in information theory? If not, what is there that's closest? or what does the mutual information lack to similarly describe what covariance does?

$\endgroup$
3
$\begingroup$

I would think that actually, covariance and mutual information share the properties of being unbounded and having units, and thus being physically interpretable.

On the other hand correlation and "normalized" mutual information -- wikipedia suggests $\frac{I(X;Y)}{\sqrt{H(X)H(Y)}}$, among a number of other options -- are both bounded (correlation between -1 and 1, and normalized mutual information between 0 and 1), and are also unitless.

tl;dr, correlation is to covariance as normalized mutual information is to mutual information.

$\endgroup$
6
  • $\begingroup$ correlation explains how variables move together, while covariance represents how variables move apart, right? How would the same statement fare for mutual information and normalized mutual information? they both seem to represent an identical concept, even though the second is just a numerical transformation of the first $\endgroup$
    – develarist
    Aug 11 '20 at 15:23
  • 1
    $\begingroup$ i would rather say -- correlation explains how variables move together (restricted to linear relationships) in relative terms, where as covariance does the same, but in absolute quantities (e.g. covariance between measurements of two calipers might be 2 mm^2, which is a physical quantity). NMI measures how much information one variable gives about another in relative terms, whereas MI measures how much information in absolute quantities (e.g. in number of bits). $\endgroup$
    – shimao
    Aug 11 '20 at 15:29
  • $\begingroup$ if mutual information (MI) is to covariance what normalized normalized information (NMI) is to correlation because NMI is bounded between 0 and 1, then the MI matrix is to the variance-covariance matrix ($\boldsymbol{\Sigma}$) what the NMI matrix is to the correlation matrix. what can be said about variance in the diagonals of $\boldsymbol{\Sigma}$ always being bounded at 0, whereas off-diagonals aren't? the diagonals of the MI matrix are entropies. Also, does the MI matrix mimic how covariance takes on negative values if a pair-wise correlation (NMI) is negative? $\endgroup$
    – develarist
    Aug 12 '20 at 3:16
  • $\begingroup$ yeah, the main distinction between "information" and "covariance/correlation" is that the the latter can either be positive or negative (two things can tend to happen together, or tend to exclude one another), whereas you can never have negative mutual information -- intuitively, knowing the value of one variable can never reduce your knowledge of another. the analogy you're trying to draw is limited, and i don't think you should read too deeply into it $\endgroup$
    – shimao
    Aug 12 '20 at 3:35
  • $\begingroup$ doesn't "information"'s inability to draw inference that X is moving against, rather than with Y, in the same way that covariance and correlation would reflect this with negative values, undermine its advantage? I'm sure non-linear interactions that MI is supposed to detect can move in both positive and negative directions. does information just indicate those directions in a different way somehow? $\endgroup$
    – develarist
    Aug 12 '20 at 3:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.