can you use mutual information to determine how much one variable tells you about another I wrote a Naive Bayes Classifier and want to be able to test how much information one variable gives about another.  The idea is to use the ones that are the most orthogonal and avoid using highly redundant variables.
My idea was to use the mutual information, but have found that it is not as clear cut as I first thought.  For example, if my variables are X = { 1, 2, 3, 4 } and Y = { 9, 10, 11, 12 }, then the mutual information would seem to be 2.  But if X = { 1, 2, 3, 4, 5, 6, 7, 8 } and Y = { 9, 10, 11, 12, 13, 14, 15, 16 }, then the mutual information is 3.  In both cases, however, if these numbers are lists (like columns in a database or spreadsheet), then X = 1 will always be paired with Y = 9 etc., meaning knowing one will always tell you the value of the other.  So if both cases have a 1 to 1 mapping between X and Y and the mutual information is different, how useful is the mutual information in determining how much you know about one variable by knowing the other?
Would some quantity like $$\frac{H(X,Y)}{H(X) + H(Y)}$$ or $$\frac{H(X,Y)}{I(X,Y)}$$ work better?  Does anyone have experience with determining how much one variable will tell you about another?
Here is a summary of the two examples.
Example 1


*

*X = { 1, 2, 3, 4 }

*Y = { 9, 10, 11, 12 }

*H(X) = 2

*H(Y) = 2

*H(X,Y) = 2

*I(X,Y) = 2

*H(X,Y) / (H(X) + H(Y)) = 0.5

*H(X,Y) / I(X,Y) = 1.0


Example 2


*

*X = { 1, 2, 3, 4, 5, 6, 7, 8 }

*Y = { 9, 10, 11, 12, 13, 14, 15, 16 }

*H(X) = 3

*H(Y) = 3 

*H(X,Y) = 3

*I(X,Y) = 3

*H(X,Y) / (H(X) + H(Y)) = 0.5

*

*H(X,Y) / I(X,Y) = 1.0


 A: Yes, you can use mutual information to determine how much one variable tells you about another – as a matter of fact, that's the exact purpose of mutual information.
The reason for the different results in the two examples is that in the second example you have more possible values for each variable, which means there is more information to possibly gain about it.
I assume you used the base-2 logarithm, such that information is measured in bits. In example 1 there are 4 possible values, each occurring equally likely, which means knowing the value of $X$ or of $Y$ constitutes an amount of information of $\log_2 4 = 2$ bits. Because there is a one-to-one-correspondence, knowing the value of $X$ means exactly knowing the value of $Y$ (and vice versa), which means the value of $X$ does not only constitute 2 bits about $X$, but also 2 bits about $Y$ (and vice versa).
In example 2 there are 8 possible values, which means knowing $X$ or $Y$ constitutes $\log_2 8 = 3$ bits, and again there is a one-to-one correspondence, so that knowing $X$ tells you 3 bits about $Y$ (and vice versa).
The formal MI result therefore makes perfect sense.
A: Relative entropy (or Kullback-Leibler divergence ) is the appropriate quantity to measure how much one variable tells about another one. Relative entropy is, unlike the mutual information, unsymmetrical. That means, it is possible that knowing a variable A tells everything about another variable B, even knowing B does not tell anything about A.
