13
$\begingroup$

I've worked with the mutual information for some time. But I found a very recent measure in the "correlation world" that can also be used to measure distribution independence, the so called "distance correlation" ( also termed Brownian correlation): http://en.wikipedia.org/wiki/Brownian_covariance. I checked the papers where this measure is introduced, but without finding any allusion to the mutual information.

So, my questions are:

  • Do they solve exactly the same problem? If not, how the problems are different?
  • And if the previous question can be answered on the positive, what are the advantages of using one or the other?
$\endgroup$
  • $\begingroup$ Try to write down explicitly 'distance correlation' and 'mutual information' for a simple example. In the second case you will get logarithms, while in the first - not. $\endgroup$ – Piotr Migdal Jan 10 '12 at 11:52
  • $\begingroup$ @PiotrMigdal Yes, I'm aware of that difference. Could you please explain why is it important? Please, take into account that I'm not a statistician... $\endgroup$ – dsign Jan 10 '12 at 11:56
  • $\begingroup$ For ma a standard tool measuring mutual dependence of probability distributions is the mutual information. It has a lot of nice properties and its interpretation is straightforward. However, there may be specific problems where distance correlation is preferred (but I have never used it in my life). So what is the problem you are trying to solve? $\endgroup$ – Piotr Migdal Jan 10 '12 at 14:53
  • 2
    $\begingroup$ This comment is a few years late but Columbia University's Statistics Dept made the academic year 2013-2014 a year of focus on measures of dependence. In April-May 2014, a workshop was held that brought together the top academics doing work in this field including the Reshef Brothers (MIC), Gabor Szekely (distance correlations), Subhadeep Mukhopadhay to name a few. Here's a link to the program that includes many pdfs from the presentations. dependence2013.wikischolars.columbia.edu/… $\endgroup$ – Mike Hunter Oct 15 '15 at 11:24
9
$\begingroup$

Information / mutual information does not depend on the possible values, it depends only on the probabilities therefore it is less sensitive. Distance correlation is more powerful and simpler to compute. For a comparision see

http://www-stat.stanford.edu/~tibs/reshef/comment.pdf

$\endgroup$
  • 2
    $\begingroup$ Hi, thanks for your answer! The paper you refer to is about MIC, which is I'm believe is a bit more than MI. I have implemented the distance correlation measure and I don't think it be simpler than the MI for the elemental case of discrete categorical variables. Then again one thing that I have learned is that DCM is well defined and well behaved for continuous variables, but with MI you need to do binning or fancy stuff ala MIC. $\endgroup$ – dsign Feb 24 '12 at 14:05
  • 2
    $\begingroup$ However, DCM seems to need square matrices whose side is the number of samples. In other words, space complexity scales quadratically. Or at least that's my impression, I would like to be in a mistake. MIC does better, because you can tune it in some sort of compromise between precision and performance. $\endgroup$ – dsign Feb 24 '12 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.