Consider three discrete binary random variables -- A, B, C.

I have calculated I(A;B) and I(A;C), but I want to calculate I(A; BC)

I'm having trouble implementing this.

From reading this paper I have an idea, but can't get it to work out in python.

I have these functions (python 3)

def entropy(X, Y): probs = [] for c1 in set(X): for c2 in set(Y): probs.append(np.mean(np.logical_and(X == c1, Y == c2))) return np.sum(-p * np.log2(p) for p in probs if p > 0)


def calc_MI(x, y, bins): c_xy = np.histogram2d(x, y, bins)[0] mi = mutual_info_score(None, None, contingency=c_xy) return mi

But I'm not sure how to move forward, thanks in advance.

  • $\begingroup$ do you mean you want to compute I(A;B|C)? $\endgroup$ – Simone May 1 '17 at 18:05
  • $\begingroup$ I don't think so, no. If you look at section 4.3 of the paper, under "Properties" of conditional mutual information, there is the chain rule. I want to use that to compute I(A; B C) $\endgroup$ – elliptic_kid May 1 '17 at 18:37

If you want to compute I(A;BC) you do not necessarily have to use the conditional mutual information. I(A;BC) is the mutual information between A and the joint variable BC.

In python you might encode BC with 4 values: 0,1,2,3 for all the combinations of values for B and C. You might do this using: 2*b + c where b can be either 0 or 1 and c the same. Then you can compute the mutual information between A and BC.

Otherwise as you point out, you can use the formula from the paper: I(A;BC) = I(A;B) + I(A;C|B). In this case you have to use conditional probabilities.

  • $\begingroup$ Thank you. Encoding the combinations of values is the key I guess. $\endgroup$ – elliptic_kid May 2 '17 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.