As for the title, the idea is to use mutual information, here and after MI, to estimate "correlation" (defined as "how much I know about A when I know B") between a continuous variable and a categorical variable. I will tell you my thoughts on the matter in a moment, but before I advice you to read this other question/answer on CrossValidated as it contains some useful information.
Now, because we cannot integrate over a categorical variable we need to discretize the continuous one. This can be done quite easily in R, which is the language I have done most of my analyses with. I preferred to use the
cut function, since it also alias the values, but other options are available as well. The point is, one has to decide a priori the number of "bins" (discrete states) before any discretization can be done.
The main problem, however, is another one: MI ranges from 0 to ∞, as it is an unstandardised measure which unit is the bit. That makes very difficult to use it as a correlation coefficient. This can be partly solved using global correlation coefficient, here and after GCC, which is a standardized version of MI; GCC is defined as follow:
Reference: the formula is from Mutual Information as a Nonlinear Tool for Analyzing Stock Market Globalization by Andreia Dionísio, Rui Menezes & Diana Mendes, 2010.
GCC ranges from 0 to 1, and therefore can easily be used to estimate the correlation between two variables. Problem solved, right? Well, kind of. Because all this process depends heavily on the number of 'bins' we decided to use during the discretization. Here the results of my experiments:
On the y-axis you have GCC and on the x-axis you have the number of 'bins' I decided to use for discretization. The two lines refers to two different analyses I conducted on two different (although very similar) datasets.
It seems to me that the usage of MI in general and GCC in particular is still controversial. Yet, this confusion may be the result of a mistake from my side. Either the case, I'd love to hear your opinion on the matter (also, do you have alternative methods to estimate correlation between a categorical variable and a continuous one?).