How to correctly compute mutual information (Python Example)? [closed]

Question

I am trying to compute mutual information for 2 vectors. I made a general function that recognizes if the data is categorical or continuous. It's really difficult to find simple examples of this calculation and I have only found theoretical implementations (e.g. How to calculate mutual information?). I have counts data that have been normalized (not integers anymore) and I want to calculate the mutual information between 2 of the rows. I am looking at the R documentation https://cran.r-project.org/web/packages/entropy/entropy.pdf and it discretizes the continuous data into bins. Why does it need to do this?

Is my implementation correct? If not, why and how can it be fixed to accurately calculate mutual information?

def shannon_entropy(A, mode="auto", verbose=False):
    """
    https://stackoverflow.com/questions/42683287/python-numpy-shannon-entropy-array
    """
    A = np.asarray(A)

    # Determine distribution type
    if mode == "auto":
        condition = np.all(A.astype(float) == A.astype(int))
        if condition:
            mode = "discrete"
        else:
            mode = "continuous"
    if verbose:
        print(mode, file=sys.stderr)
    # Compute shannon entropy
    pA = A / A.sum()
    # Remove zeros
    pA = pA[np.nonzero(pA)[0]]
    if mode == "continuous":
        return -np.sum(pA*np.log2(A))  
    if mode == "discrete":
        return -np.sum(pA*np.log2(pA))   

def mutual_information(x,y, mode="auto", normalized=False):
    """
    I(X, Y) = H(X) + H(Y) - H(X,Y)
    https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy
    """
    x = np.asarray(x)
    y = np.asarray(y)
    # Determine distribution type
    if mode == "auto":
        condition_1 = np.all(x.astype(float) == x.astype(int))
        condition_2 = np.all(y.astype(float) == y.astype(int))
        if all([condition_1, condition_2]):
            mode = "discrete"
        else:
            mode = "continuous"

    H_x = shannon_entropy(x, mode=mode)
    H_y = shannon_entropy(y, mode=mode)
    H_xy = shannon_entropy(np.concatenate([x,y]), mode=mode)

    # Mutual Information
    I_xy = H_x + H_y - H_xy
    if normalized:
        return I_xy/np.sqrt(H_x*H_y)
    else:
        return  I_xy

Answer 1

To calculate mutual information, you need to know the distribution of the pair $(X,Y)$ which is counts for each possible value of the pair. This would be described by a 2 dimensional matrix as in https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy.

Instead you have two one dimensional count vectors as arguments, that is you only know the marginal distributions. Computing the mutual information of two distributions does not make sense. You can only compute the mutual information of a joint distribution (=distribution of the pair).