# Conditional Mutual Information, Chain Rule

Consider three discrete binary random variables -- A, B, C.

I have calculated I(A;B) and I(A;C), but I want to calculate I(A; BC)

I'm having trouble implementing this.

From reading this paper I have an idea, but can't get it to work out in python.

I have these functions (python 3)

def entropy(X, Y): probs = [] for c1 in set(X): for c2 in set(Y): probs.append(np.mean(np.logical_and(X == c1, Y == c2))) return np.sum(-p * np.log2(p) for p in probs if p > 0)

and

def calc_MI(x, y, bins): c_xy = np.histogram2d(x, y, bins) mi = mutual_info_score(None, None, contingency=c_xy) return mi

But I'm not sure how to move forward, thanks in advance.

• do you mean you want to compute I(A;B|C)? – Simone May 1 '17 at 18:05
• I don't think so, no. If you look at section 4.3 of the paper, under "Properties" of conditional mutual information, there is the chain rule. I want to use that to compute I(A; B C) – elliptic_kid May 1 '17 at 18:37

If you want to compute I(A;BC) you do not necessarily have to use the conditional mutual information. I(A;BC) is the mutual information between A and the joint variable BC.
In python you might encode BC with 4 values: 0,1,2,3 for all the combinations of values for B and C. You might do this using: 2*b + c where b can be either 0 or 1 and c the same. Then you can compute the mutual information between A and BC.
Otherwise as you point out, you can use the formula from the paper: I(A;BC) = I(A;B) + I(A;C|B). In this case you have to use conditional probabilities.