I am trying to compute mutual information for 2 vectors. I made a general function that recognizes if the data is categorical or continuous. It's really difficult to find simple examples of this calculation and I have only found theoretical implementations (e.g. How to calculate mutual information?). I have counts data that have been normalized (not integers anymore) and I want to calculate the mutual information between 2 of the rows. I am looking at the R documentation https://cran.r-project.org/web/packages/entropy/entropy.pdf and it discretizes the continuous data into bins. Why does it need to do this?
Is my implementation correct? If not, why and how can it be fixed to accurately calculate mutual information?
def shannon_entropy(A, mode="auto", verbose=False): """ https://stackoverflow.com/questions/42683287/python-numpy-shannon-entropy-array """ A = np.asarray(A) # Determine distribution type if mode == "auto": condition = np.all(A.astype(float) == A.astype(int)) if condition: mode = "discrete" else: mode = "continuous" if verbose: print(mode, file=sys.stderr) # Compute shannon entropy pA = A / A.sum() # Remove zeros pA = pA[np.nonzero(pA)] if mode == "continuous": return -np.sum(pA*np.log2(A)) if mode == "discrete": return -np.sum(pA*np.log2(pA)) def mutual_information(x,y, mode="auto", normalized=False): """ I(X, Y) = H(X) + H(Y) - H(X,Y) https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy """ x = np.asarray(x) y = np.asarray(y) # Determine distribution type if mode == "auto": condition_1 = np.all(x.astype(float) == x.astype(int)) condition_2 = np.all(y.astype(float) == y.astype(int)) if all([condition_1, condition_2]): mode = "discrete" else: mode = "continuous" H_x = shannon_entropy(x, mode=mode) H_y = shannon_entropy(y, mode=mode) H_xy = shannon_entropy(np.concatenate([x,y]), mode=mode) # Mutual Information I_xy = H_x + H_y - H_xy if normalized: return I_xy/np.sqrt(H_x*H_y) else: return I_xy