Let's assume we have three continuous or discrete variables, X, Y and Z, for which I want to calculate Mutual Information (MI) and Conditional Mutual Information (CMI). The formulas to calculate this through Shannon Entropy (H) of the distributions for them would be:
MI(X;Y) = H(X)+H(Y)-H(X,Y)
MI(X;Z) = H(X)+H(Z)-H(X,Z)
CMI(X;Y|Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)
My question is, are these quantities directly comparable? In an operation like:
CMI(X;Y|Z)-MI(X;Y)+MI(X;Z)
If they are not directly comparable, can this be converted into the following formula?
CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)-H(X)-H(Y)+H(X,Y)+H(X)+H(Z)-H(X,Z)
That simplifies into:
CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = H(X,Y)+H(Y,Z)-H(X,Y,Z)-H(Y)
Which brings us to the following conclusion:
CMI(X;Y|Z)-MI(X;Y)+MI(X;Z) = CMI(X;Z|Y)
Thank you for any tip!