Interpretation of mutual information Is it correct to say that the mutual information $I(X;Y)$ quantifies how well we can discriminate among the outcomes of $Y$ by looking at the outcomes of $X$ (or viceversa)? If yes, do you have any reference that uses this interpretation?
Further explanation: I am not looking for a explanation of the meaning of MI. I am interested specifically in the interpretation that I wrote above. I remember having read it somewhere, but cannot remember what the source was. This interpretation emphasizes the role of discrimination in the definition of MI. The concept of discrimination is ingrained in the definition of mutual information through the following identity: if $Y$ has only one possible outcome, then the mutual information between $X$ and $Y$ is identically zero. For my work this interpretation it is currently very important and I would like to be able to cite a source.
 A: The mutual information measure $I(X;Y)$ is nonparametric measure of probabilistic dependence between the variables $X$ and $Y$. As follows from wikipedia:
"Intuitively, mutual information measures the information that $X$ and $Y$ share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if $X$  and $Y$ are independent, then knowing $X$ does not give any information about $Y$ and vice versa, so their mutual information is zero."
In general, $I(X;Y)$ is computed for $m \times 2$ grid-histograms. You can 'bin' continuously distributed variables into $m$ intervals as to create this grid.
When it comes to the degree of covariation between a feature value distribution and a class outcome distribution, the information gain $IG(T,a)$ is widely used. Here $T$ is the variable associated with class outcomes and $a$ the attribute value. I refer you to the definition of criteria optimized by learning algorithm ID3 (its modern successor algorithm is called C4.5). $IG(T,a)$ is different to $I(T;A)$.
$I(X;Y)$ is also defined for continuous probability density functions, but you need to know the mathematical formula for the bivariate probability density in order to calculate it. Hence, histograms are practical for continuous stochastic variables $X$ and $Y$.
