13
$\begingroup$

I am trying to compute mutual information for 2 vectors. I made a general function that recognizes if the data is categorical or continuous. It's really difficult to find simple examples of this calculation and I have only found theoretical implementations (e.g. How to calculate mutual information?). I have counts data that have been normalized (not integers anymore) and I want to calculate the mutual information between 2 of the rows. I am looking at the R documentation https://cran.r-project.org/web/packages/entropy/entropy.pdf and it discretizes the continuous data into bins. Why does it need to do this?

Is my implementation correct? If not, why and how can it be fixed to accurately calculate mutual information?

def shannon_entropy(A, mode="auto", verbose=False):
    """
    https://stackoverflow.com/questions/42683287/python-numpy-shannon-entropy-array
    """
    A = np.asarray(A)

    # Determine distribution type
    if mode == "auto":
        condition = np.all(A.astype(float) == A.astype(int))
        if condition:
            mode = "discrete"
        else:
            mode = "continuous"
    if verbose:
        print(mode, file=sys.stderr)
    # Compute shannon entropy
    pA = A / A.sum()
    # Remove zeros
    pA = pA[np.nonzero(pA)[0]]
    if mode == "continuous":
        return -np.sum(pA*np.log2(A))  
    if mode == "discrete":
        return -np.sum(pA*np.log2(pA))   

def mutual_information(x,y, mode="auto", normalized=False):
    """
    I(X, Y) = H(X) + H(Y) - H(X,Y)
    https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy
    """
    x = np.asarray(x)
    y = np.asarray(y)
    # Determine distribution type
    if mode == "auto":
        condition_1 = np.all(x.astype(float) == x.astype(int))
        condition_2 = np.all(y.astype(float) == y.astype(int))
        if all([condition_1, condition_2]):
            mode = "discrete"
        else:
            mode = "continuous"

    H_x = shannon_entropy(x, mode=mode)
    H_y = shannon_entropy(y, mode=mode)
    H_xy = shannon_entropy(np.concatenate([x,y]), mode=mode)

    # Mutual Information
    I_xy = H_x + H_y - H_xy
    if normalized:
        return I_xy/np.sqrt(H_x*H_y)
    else:
        return  I_xy
$\endgroup$
2
  • $\begingroup$ Why are you doing "x.astype(float) == x.astype(int)"? You seem to be confusing values and counts of values. Why are you concatenating x and y? Your formula for continuous entropy is wrong, although off by an amount that will cancel out when doing mutual information. As for discretizing the data, if you're estimating the probs from empirical data, you need multiple instances of a value to calculate the prob for that value. The probability of getting the exact same value twice from a continuous distribution is zero. Therefore, your calculated entropy will always be log(size of sample). $\endgroup$ Commented Oct 3, 2017 at 21:16
  • $\begingroup$ I was trying to do an automated implementation of what was suggested in: stackoverflow.com/questions/42683287/…. Does that mean you need to fit the count data to a distribution? $\endgroup$
    – O.rka
    Commented Oct 3, 2017 at 21:53

1 Answer 1

17
$\begingroup$

To calculate mutual information, you need to know the distribution of the pair $(X,Y)$ which is counts for each possible value of the pair. This would be described by a 2 dimensional matrix as in https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy.

Instead you have two one dimensional count vectors as arguments, that is you only know the marginal distributions. Computing the mutual information of two distributions does not make sense. You can only compute the mutual information of a joint distribution (=distribution of the pair).

$\endgroup$
2
  • $\begingroup$ Thanks for responding. Is this why the values need to be discretized? I'm running the example in the link you sent and it's difficult to know what value to use for the amount of bins to split the data into. It seems to increase linearly. i.imgur.com/QENI6Sf.png $\endgroup$
    – O.rka
    Commented Oct 3, 2017 at 22:04
  • $\begingroup$ Data must be discretized because this kind of entropy estimation needs to have most often several observations for each bin to give a realistic estimate. But your computation for the moment does nothing meaningful so that what you observe does not help. You must first implement a true computation and then work on the problem of discretization. $\endgroup$ Commented Oct 4, 2017 at 8:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.