# How to correctly compute mutual information (Python Example)? [closed]

I am trying to compute mutual information for 2 vectors. I made a general function that recognizes if the data is categorical or continuous. It's really difficult to find simple examples of this calculation and I have only found theoretical implementations (e.g. How to calculate mutual information?). I have counts data that have been normalized (not integers anymore) and I want to calculate the mutual information between 2 of the rows. I am looking at the R documentation https://cran.r-project.org/web/packages/entropy/entropy.pdf and it discretizes the continuous data into bins. Why does it need to do this?

Is my implementation correct? If not, why and how can it be fixed to accurately calculate mutual information?

def shannon_entropy(A, mode="auto", verbose=False):
"""
https://stackoverflow.com/questions/42683287/python-numpy-shannon-entropy-array
"""
A = np.asarray(A)

# Determine distribution type
if mode == "auto":
condition = np.all(A.astype(float) == A.astype(int))
if condition:
mode = "discrete"
else:
mode = "continuous"
if verbose:
print(mode, file=sys.stderr)
# Compute shannon entropy
pA = A / A.sum()
# Remove zeros
pA = pA[np.nonzero(pA)[0]]
if mode == "continuous":
return -np.sum(pA*np.log2(A))
if mode == "discrete":
return -np.sum(pA*np.log2(pA))

def mutual_information(x,y, mode="auto", normalized=False):
"""
I(X, Y) = H(X) + H(Y) - H(X,Y)
https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy
"""
x = np.asarray(x)
y = np.asarray(y)
# Determine distribution type
if mode == "auto":
condition_1 = np.all(x.astype(float) == x.astype(int))
condition_2 = np.all(y.astype(float) == y.astype(int))
if all([condition_1, condition_2]):
mode = "discrete"
else:
mode = "continuous"

H_x = shannon_entropy(x, mode=mode)
H_y = shannon_entropy(y, mode=mode)
H_xy = shannon_entropy(np.concatenate([x,y]), mode=mode)

# Mutual Information
I_xy = H_x + H_y - H_xy
if normalized:
return I_xy/np.sqrt(H_x*H_y)
else:
return  I_xy
• Why are you doing "x.astype(float) == x.astype(int)"? You seem to be confusing values and counts of values. Why are you concatenating x and y? Your formula for continuous entropy is wrong, although off by an amount that will cancel out when doing mutual information. As for discretizing the data, if you're estimating the probs from empirical data, you need multiple instances of a value to calculate the prob for that value. The probability of getting the exact same value twice from a continuous distribution is zero. Therefore, your calculated entropy will always be log(size of sample). Commented Oct 3, 2017 at 21:16
• I was trying to do an automated implementation of what was suggested in: stackoverflow.com/questions/42683287/…. Does that mean you need to fit the count data to a distribution? Commented Oct 3, 2017 at 21:53

To calculate mutual information, you need to know the distribution of the pair $(X,Y)$ which is counts for each possible value of the pair. This would be described by a 2 dimensional matrix as in https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy.