I've seen a couple talks by non-statisticians where they seem to reinvent correlation measures using mutual information rather than regression (or equivalent/closely-related statistical tests).

I take it there's a good reason statisticians don't take this approach. My layman's understanding is that estimators of entropy / mutual information tend to be problematic and unstable. I assume power is also problematic as a result: they try to get around this by claiming that they're not using a parametric testing framework. Usually this kind of work doesn't bother with power calculations, or even confidence/credible intervals.

But to take a devil's advocate position, is slow convergence that big of a deal when datasets are extremely large? Also, sometimes these methods seem to "work" in the sense that the associations are validated by follow-up studies. What's the best critique against using mutual information as a measure of association and why isn't it widely used in statistical practice?

edit: Also, are there any good papers that cover these issues?

  • 3
    $\begingroup$ MI is a measure of association between two discrete variables. It's not really that common a setting in general statistics (could be in some specialized sub fields). But within that setting, I see it used frequently enough. Certainly, when I encounter applied people using Pearson correlation on bivariate discrete datasets, I point out MI to them. $\endgroup$ – user603 Aug 22 '13 at 3:00
  • 1
    $\begingroup$ See also stats.stackexchange.com/questions/1052/… However, the discussion here is already, in my view, as good or better, so the usual question about duplicates is moot. $\endgroup$ – Nick Cox Aug 22 '13 at 9:33
  • $\begingroup$ Also for references see stats.stackexchange.com/q/20011/1036 $\endgroup$ – Andy W Aug 22 '13 at 11:56
  • 2
    $\begingroup$ A further general reference is Matthew Reimherr and Dan L. Nicolae. 2013. On Quantifying Dependence: A Framework for Developing Interpretable Measures. Statistical Science 28: 116-130. $\endgroup$ – Nick Cox Aug 23 '13 at 11:03

I think you should distinguish between categorical (discrete) data and continuous data.

For continuous data, Pearson correlation measures a linear (monotonic) relationship, rank correlation a monotonic relationship.

MI on the other hand "detects" any relationship. This is normally not what you are interested in and/or is likely to be noise. In particular, you have to estimate the density of the distribution. But since it is continuous, you would first create a histogram [discrete bins], and then calculate MI. But since MI allows for any relationship, the MI will change as you use smaller bins (i.e. so you allow more wiggles). So you can see that the estimation of MI will be very unstable, not allowing you to put any confidence intervals on the estimate etc. [Same goes if you do a continuous density estimate.] Basically there are too many things to estimate before actually calculating the MI.

Categorical data on the other hand fits quite nicely into MI framework (see G-test), and there is not much to choose between G-test and chi-squared.

  • $\begingroup$ I'm mostly referring to cases of discrete association (by regression, I had GLMs in mind, not just OLS). Actually, a lot of scientists studying complex phenomena (e.g. genetics) might say they are more interested in just what you're describing (detect any relationship). The lure of escaping the obvious common criticism of "what if the functional form of the correlation is wrong? Of course I want to detect any relationship!" is strong. However, I think there's a no-free-lunch fallacy at play here, but that would be being overlooked that I'm trying to better articulate/understand. $\endgroup$ – user4733 Aug 22 '13 at 12:24
  • 1
    $\begingroup$ ... I was not aware of the the relationship between LR tests and MI though, that's very interesting! $\endgroup$ – user4733 Aug 22 '13 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.