# Estimating mutual information between two processes

I have pairs of two signals (with measurement noises) and wonder what would be the best way to test if those two signals are correlated or not, in a rather model-free, nonparametric way. If I pretend to be a bit more formal, I wonder how to estimate the mutual information between two random processes from sample pairs.

• If you can "line up" the signals so that you have a list of pairs of measurements (one from signal 1, one from signal 2), then you could use something simple like Spearman's rank correlation coefficient to test for independence. Or just plot one signal against the other and let your eyes do the work. I'm not sure mutual information (the concept related to entropy) is really the way to go here. Commented Sep 6, 2015 at 6:58
• Thanks Creosote for your help. I like your suggestion but the data I am looking is too complicated to find their correlations by eye or Spearman's test. Thank. Commented Sep 17, 2015 at 12:38

Estimating the mutual information makes sense as a general and model-free way to measure the relationship between two variables. The mutual information will be zero if and only if the two signals are independent. The magnitude of the mutual information can be interpreted as the number of bits you save by compressing two signals, x and y, together rather than compressing them separately (that's just one interpretation, for more way of thinking about this you can see, e.g., Cover & Thomas's Information Theory book).

However, there are still issues to consider. First of all, how do you estimate mutual information at all? Mutual information is integral of p(x,y) log [p(x,y)/(p(x) p(y))]. For continuous variables (I assume that is what your question is about), you have a few options.

1. Estimate the probability distribution according to some parametric model, then calculate the mutual information analytically (i.e. do the integral).
2. Discretize the data and then use the formula for mutual information for discrete variables.
3. Use kernel density estimation and then do the integral analytically.
4. Use non-parametric entropy estimators.

The drawback of 1 is that it requires model assumptions which you seem to want to avoid. The drawback of number 2 is that you have to decide how to discretize the data (how many bins and how wide should they be?) and your result will be very sensitive to these choices (see this paper). Number 3 is a great choice, especially if you are using 1-dimensional data. I'm not an expert on this so I can't provide links. The only choice you have to make is the "bandwidth", but if you search the literature you'll find rules of thumb for this.

I'm a fan of number 4 which is to use estimators related to Kraskov et. al's method (paper). These approaches do not make assumptions about the underlying distribution but will asymptotically (in the large data limit) converge to the true mutual information. This sounds perfect! What can go wrong? If the data is very close to a manifold, for instance, these estimators do quite badly and require a lot of data to converge (paper). Options implementing these estimators exist in Java and python. Also check out Barnabas Poczos work, including code for estimators and theoretical properties of different estimation procedures.

• Thanks Greg for your wonderful summary and suggestions. I will take a look at the papers on number 4. Meanwhile, I came up with a toolbox from Alessandro Montalto (mutetoolbox.guru) that implements Kraskov's method as one of the entropy estimators. Thanks again! Commented Sep 17, 2015 at 12:47