I have pairs of two signals (with measurement noises) and wonder what would be the best way to test if those two signals are correlated or not, in a rather model-free, nonparametric way. If I pretend to be a bit more formal, I wonder how to estimate the mutual information between two random processes from sample pairs.
Estimating the mutual information makes sense as a general and model-free way to measure the relationship between two variables. The mutual information will be zero if and only if the two signals are independent. The magnitude of the mutual information can be interpreted as the number of bits you save by compressing two signals, x and y, together rather than compressing them separately (that's just one interpretation, for more way of thinking about this you can see, e.g., Cover & Thomas's Information Theory book).
However, there are still issues to consider. First of all, how do you estimate mutual information at all? Mutual information is integral of p(x,y) log [p(x,y)/(p(x) p(y))]. For continuous variables (I assume that is what your question is about), you have a few options.
- Estimate the probability distribution according to some parametric model, then calculate the mutual information analytically (i.e. do the integral).
- Discretize the data and then use the formula for mutual information for discrete variables.
- Use kernel density estimation and then do the integral analytically.
- Use non-parametric entropy estimators.
The drawback of 1 is that it requires model assumptions which you seem to want to avoid. The drawback of number 2 is that you have to decide how to discretize the data (how many bins and how wide should they be?) and your result will be very sensitive to these choices (see this paper). Number 3 is a great choice, especially if you are using 1-dimensional data. I'm not an expert on this so I can't provide links. The only choice you have to make is the "bandwidth", but if you search the literature you'll find rules of thumb for this.
I'm a fan of number 4 which is to use estimators related to Kraskov et. al's method (paper). These approaches do not make assumptions about the underlying distribution but will asymptotically (in the large data limit) converge to the true mutual information. This sounds perfect! What can go wrong? If the data is very close to a manifold, for instance, these estimators do quite badly and require a lot of data to converge (paper). Options implementing these estimators exist in Java and python. Also check out Barnabas Poczos work, including code for estimators and theoretical properties of different estimation procedures.