3
$\begingroup$

Is it correct to say that the mutual information $I(X;Y)$ quantifies how well we can discriminate among the outcomes of $Y$ by looking at the outcomes of $X$ (or viceversa)? If yes, do you have any reference that uses this interpretation?

Further explanation: I am not looking for a explanation of the meaning of MI. I am interested specifically in the interpretation that I wrote above. I remember having read it somewhere, but cannot remember what the source was. This interpretation emphasizes the role of discrimination in the definition of MI. The concept of discrimination is ingrained in the definition of mutual information through the following identity: if $Y$ has only one possible outcome, then the mutual information between $X$ and $Y$ is identically zero. For my work this interpretation it is currently very important and I would like to be able to cite a source.

$\endgroup$

1 Answer 1

2
$\begingroup$

The mutual information measure $I(X;Y)$ is nonparametric measure of probabilistic dependence between the variables $X$ and $Y$. As follows from wikipedia:

"Intuitively, mutual information measures the information that $X$ and $Y$ share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if $X$ and $Y$ are independent, then knowing $X$ does not give any information about $Y$ and vice versa, so their mutual information is zero."

In general, $I(X;Y)$ is computed for $m \times 2$ grid-histograms. You can 'bin' continuously distributed variables into $m$ intervals as to create this grid.

When it comes to the degree of covariation between a feature value distribution and a class outcome distribution, the information gain $IG(T,a)$ is widely used. Here $T$ is the variable associated with class outcomes and $a$ the attribute value. I refer you to the definition of criteria optimized by learning algorithm ID3 (its modern successor algorithm is called C4.5). $IG(T,a)$ is different to $I(T;A)$.

$I(X;Y)$ is also defined for continuous probability density functions, but you need to know the mathematical formula for the bivariate probability density in order to calculate it. Hence, histograms are practical for continuous stochastic variables $X$ and $Y$.

$\endgroup$
9
  • 1
    $\begingroup$ Careful...mutual information isn’t restricted to describing linear or even monotonic relationships. $\endgroup$
    – Dave
    Jul 26, 2020 at 15:06
  • 1
    $\begingroup$ Thanks! - I have added a comment on that in the answer text. $\endgroup$ Jul 26, 2020 at 15:24
  • $\begingroup$ “Covariance” also refers to a linear relationship, and your terminology is not quite that but very close. You could drop that first paragraph (still credit Wiki) and have a correct answer. Also, your last paragraph doesn’t make sense to me. It’s nonparametric because you can shuffle the rows...huh? $\endgroup$
    – Dave
    Jul 26, 2020 at 15:30
  • $\begingroup$ See the text now. I instead use the terms probabilistic dependence. $\endgroup$ Jul 26, 2020 at 15:32
  • $\begingroup$ I like the expression "probabilistic dependence". Very elegant! $\endgroup$
    – Cesare
    Jul 26, 2020 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.