3
$\begingroup$

I'm trying to calculate mutual information in Python, using numpy. My implementation so far is:

def mutual_info(x, y, bins=[10000, 10000]):
    """Calculate mutual information based on 2D histrograms
    """
    hist = np.histogram2d(x, y, bins, normed=True)[0]
    joint_prob = hist / np.sum(hist)

    hist = np.histogram(x, bins[0], density=True)[0]
    probs_x = hist / np.sum(hist)

    hist = np.histogram(y, bins[1], density=True)[0]
    probs_y = hist / np.sum(hist)

    probs = joint_prob / (np.reshape(probs_x, [-1, 1]) * probs_y)

    # use masked array to avoid NaNs
    info = (joint_prob * np.ma.log2(probs)).sum()

    return info

The problem is that it is not returning a near-zero value for two completely independent random variables:

a = np.random.rand(10000)
b = np.random.rand(10000)
mutual_info(a, a)
# 12.471484491681876
mutual_info(a, b)
# 11.640764212276384

I notice that in R, using the mi.plugin function from the entropy package also doesn't result in a near-zero result, but it is at least much lower for independent variables:

R> a = runif(10000)
R> b = runif(10000)
R> counts = hist2d(a, a, show=FALSE, bins=1000)[['counts']]
R> mi.plugin(counts)
[1] 5.288
R> counts = hist2d(a, b, show=FALSE, bins=1000)[['counts']]R> mi.plugin(counts)
[1] 1.532

Is there something wrong with my implementation, or am I misunderstanding mutual information?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ What is the special need for binning here? Binning causes all types of problems. At first glance this appears to call for a rank correlation association measure. $\endgroup$
    – Frank Harrell
    Commented Jan 27, 2016 at 12:42
  • $\begingroup$ @FrankHarrell: efficiency - KDE methods seem much slower, and I want to apply this to a lot of data, for feature selection for a neural network. I'm not familiar with rank correlation, and I'm not sure if it's related to what I'm looking for. $\endgroup$
    – naught101
    Commented Jan 27, 2016 at 22:01
  • $\begingroup$ @FrankHarrell: Ahh - rank correlation is for discrete data, right? I'm looking at continuous data. $\endgroup$
    – naught101
    Commented Jan 27, 2016 at 23:13
  • 2
    $\begingroup$ On the contrary rank correlation is for ordinal and binary data. Continuous variables are also ordinal, and rank correlations are highly efficient for continuous data while making fewer assumptions. There are extensions of rank correlation measures/tests if you want to detect non-monotonic relationships. $\endgroup$
    – Frank Harrell
    Commented Jan 28, 2016 at 12:32

1 Answer 1

Reset to default
5
$\begingroup$

Mutual information estimated via binning is very sensitive to the number of bins $r$ for $X$, number of bins $c$ for $Y$, and the number of data points $n$. Miller's result from 1955 states that the magnitude of the bias is [Reference]: $$ \frac{rc -1}{2n}. $$ This to say that you should not choose too many bins. There are some rule of thumb for the number of bins when estimating mutual information.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yep, that looks like about it. When I use that rule of thumb I get much more reasonable looking results. The actual MI value seems highly dependent on the number of bins and also the sample size... is that just a fact, or is it just a quantity that's difficult to calculate properly? $\endgroup$
    – naught101
    Commented Jan 27, 2016 at 10:55
  • $\begingroup$ It is due the estimation error when computing each probability by the formula $\frac{ \text{n. points for a bin}}{n}$. In practice, if you are ranking values of mutual information estimated by fixing $r$ and $c$ on a sample $n$, that bias cancels out. That's why you can still use this approach for example for feature selection. There are other estimators of mutual information you might want to try that are not based on binning: the $k$NN and the kernel based. $\endgroup$
    – Simone
    Commented Jan 27, 2016 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.