Comparable Mutual Information and Conditional Mutual Information values

Question

Let's assume we have three continuous or discrete variables, X, Y and Z, for which I want to calculate Mutual Information (MI) and Conditional Mutual Information (CMI). The formulas to calculate this through Shannon Entropy (H) of the distributions for them would be:

MI(X;Y) = H(X)+H(Y)-H(X,Y)
MI(X;Z) = H(X)+H(Z)-H(X,Z)
CMI(X;Y|Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)

My question is, are these quantities directly comparable? In an operation like:

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z)

If they are not directly comparable, can this be converted into the following formula?

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)-H(X)-H(Y)+H(X,Y)+H(X)+H(Z)-H(X,Z)

That simplifies into:

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = H(X,Y)+H(Y,Z)-H(X,Y,Z)-H(Y)

Which brings us to the following conclusion:

CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = CMI(X;Z|Y)

Thank you for any tip!

Answer 1

I'm assuming by "comparable" you mean something like "you are allowed to add and subtract them" or "these have the same units."

If so, the answer is yes, as long as you use the same logarithm base when you compute the various MI/CMI values. (The units of entropy are generally called "bits" or "nats", or less commonly "shannons" or "hartleys," depending on the base of the logarithm--respectively 2, $e$, 2, and 10.)

Many similar identities are proven or assigned as exercises, for instance, in MacKay's Information Theory, Inference, and Learning Algorithms (Chapter 8), which I highly recommend if you want more intuition for how various entropies behave.

The question of whether estimators of these quantities are comparable is a tougher one, as estimating entropies and conditional entropies can be rather tricky. There are only a couple caveats to this in the estimation setting:

• This identity holds for the true mutual information/entropy, but I don't believe it necessarily holds for estimators (it depends on how you're estimating).

  If you're estimating each MI and CMI as the sum of a bunch of individual entropies, then the identity will hold since the equations go through just the same way for the estimated CMI as for the true CMI. But if you're directly estimating the (C)MI some other way, then you will need to analyze your specific estimator.

• Estimators of different mutual informations/entropies may not be independent, so be careful what assumptions you make about the variance structure of their sums and differences.

• Additionally, all estimators of mutual information and entropy are biased, so you need to study the bias properties of their sums and differences as well.