0
$\begingroup$

For a mutual information of two continuous variables (X and Y), I interpret a value of M.I. = 1 bit (i.e., 2^1 = 2 distinguishable levels), to mean the following:

If I know any given value of X, I should expect to be able to distinguish between two values of Y. Is this the correct interpretation? I'm wondering if it should really be: if I know two given values of X, and the range of potential values of X, I should be able to distinguish between values of Y given the two values of X fall far apart from each other (with respect to their entire distribution).

Further, I'm confused how I would interpret a value of M.I. = 0.5 bits, (i.e. 2^0.5, 1.4 distinguishable levels) Does this mean that it is quite unreliable to even be able to use X to distinguish two values of Y? How unreliable though?

Thanks!

$\endgroup$
2
  • $\begingroup$ Why can't you distinguish between two values of $Y$ by comparing them directly? $\endgroup$
    – jbowman
    Commented Sep 5, 2023 at 16:40
  • $\begingroup$ I'm unsure if the unidirectional interpretation you're assuming is quite the way you want to go about things? Since mutual information is symmetric, i.e., $I(X,Y) = I(Y,X)$ meaning that it is a quantification of both the amount of information obtained about $Y$ from observing $X$ and about $X$ from observing $Y$. $\endgroup$
    – QMath
    Commented Sep 8, 2023 at 2:24

1 Answer 1

1
+50
$\begingroup$

Mutual information tells you how much is uncertainty in one random variable reduced if you know the value of the other. When the mutual information is 1 bit, it means that knowing one variable allows you to classify the other into one of two bins, with no uncertainty. If the mutual information is less than one, you won't be able to perfectly classify it; you'll be occasionally making mistakes.

Example

Let $X$ and $Y$ be two random variables. They are unrelated, except for one rule: if $X$ is positive, $Y$ is positive, too, and vv. This is one bit of information:

1-bit mutual information

Compare this to another setup, where if $X$ is positive, $Y$ tends to be positive, but this is not certain (deterministic). In this case, knowing the value of $X$ allows you to make some probabilistic prediction about $Y$, better than without knowing it, but it does not suffice to decide with certainty. The mutual information is less than one bit:

mutual information < 1 bit

Note that the relationship between the variables need not be as obvious as in the above examples. In the example below, the mutual information between $X$ and $Y$ is still 1 bit:

non-obvious 1 bit mutual information

Here, $Y$ is positive if, in the decimal representation, the second decimal digit of $X$ is under 5 (e.g. for $x = 0.141$), and negative otherwise (e.g. for $x = 0.173$).

I don't know whether this counts as a reputable source, but I find it pretty clear.

$\endgroup$
2
  • $\begingroup$ Small thing first, $I(X, Y) = 1$, because you are using log base 2. Bigger question: Wouldn't differently sized bins change the result a lot? Let's say $Y < -0.5 \implies X < -0.5$. I get 0.189 (0.25 * log(0.5, base = 2) + log(1.5, base = 2) * 0.75 ) $\endgroup$ Commented Sep 11, 2023 at 8:36
  • $\begingroup$ Good point. To my understanding, however, the OP wanted to know the opposite direction. Having fixed $I(X, Y) = 1$, the two bins will be equally sized. But, true, there is no guarantee that they will be neatly aligned with coordinate axes, as in my top example. $\endgroup$
    – Igor F.
    Commented Sep 12, 2023 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.