4
$\begingroup$

Let's assume we have three continuous or discrete variables, X, Y and Z, for which I want to calculate Mutual Information (MI) and Conditional Mutual Information (CMI). The formulas to calculate this through Shannon Entropy (H) of the distributions for them would be:

MI(X;Y) = H(X)+H(Y)-H(X,Y)
MI(X;Z) = H(X)+H(Z)-H(X,Z)
CMI(X;Y|Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)

My question is, are these quantities directly comparable? In an operation like:

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z)

If they are not directly comparable, can this be converted into the following formula?

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)-H(X)-H(Y)+H(X,Y)+H(X)+H(Z)-H(X,Z)

That simplifies into:

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = H(X,Y)+H(Y,Z)-H(X,Y,Z)-H(Y)

Which brings us to the following conclusion:

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = CMI(X;Z|Y)

Thank you for any tip!

$\endgroup$
4
+100
$\begingroup$

I'm assuming by "comparable" you mean something like "you are allowed to add and subtract them" or "these have the same units."

If so, the answer is yes, as long as you use the same logarithm base when you compute the various MI/CMI values. (The units of entropy are generally called "bits" or "nats", or less commonly "shannons" or "hartleys," depending on the base of the logarithm--respectively 2, $e$, 2, and 10.)

Many similar identities are proven or assigned as exercises, for instance, in MacKay's Information Theory, Inference, and Learning Algorithms (Chapter 8), which I highly recommend if you want more intuition for how various entropies behave.

The question of whether estimators of these quantities are comparable is a tougher one, as estimating entropies and conditional entropies can be rather tricky. There are only a couple caveats to this in the estimation setting:

  • This identity holds for the true mutual information/entropy, but I don't believe it necessarily holds for estimators (it depends on how you're estimating).

    If you're estimating each MI and CMI as the sum of a bunch of individual entropies, then the identity will hold since the equations go through just the same way for the estimated CMI as for the true CMI. But if you're directly estimating the (C)MI some other way, then you will need to analyze your specific estimator.

  • Estimators of different mutual informations/entropies may not be independent, so be careful what assumptions you make about the variance structure of their sums and differences.

  • Additionally, all estimators of mutual information and entropy are biased, so you need to study the bias properties of their sums and differences as well.

$\endgroup$
  • $\begingroup$ Thank you! I would vote your answer up but I spent all my reputation for this bounty :-) You are right: the tricks are two. First, estimators may differ (however, I am using the same algorithm for MI and CMI, i.e. adaptive partitioning). Second, one must use the same logarithm base for estimating the entropies. $\endgroup$ – Federico Giorgi Feb 6 '15 at 23:30
  • 1
    $\begingroup$ Ah, I should have realized you were asking specifically about estimation; sorry. I've fleshed out the part about estimation above. I hope it's helpful. $\endgroup$ – Ben Kuhn Feb 7 '15 at 3:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.