Mutual Information for unordered variables

Question

Like many others I am seeking for a method to create a "correlation" like matrix for an unpredictable mix of variables which can contain unordered categorical ones as well.

While there are ideas of defaulting to different options depending on variable types for each pair of variables this seemed complicated (plus Spearman bears the risk for beeing computationally too expensive and I can't use Pearson given non-normality) and using e.g. polycor relies on ordered categorical variables I felt going down the mutual information route would be more straight forward as suggested here.

So I made a little example to check whether it gives me what I want. Some categorical variables, some numerical ones and you can see that both cat1 vs. cat2 as well as num1 vs. num2 are "perfectly correlated" whereas cat1 vs. cat3 as well as num1 vs. num3 are "perfectly anti-correlated", at least when you read an order into the categorical variables based on their alphabetical sort order of values (which is not what I want, see title!):

library(infotheo)

cat1 <- c('a','b','c','a','b','c','a','b','c')
cat2 <- c('x','y','z','x','y','z','x','y','z')
cat3 <- c('z','y','x','z','y','x','z','y','x')
cat4 <- c('d','e','f','d','e','e','d','d','f')
cat5 <- c('c','e','y','b','f','x',NA,NA,NA)
num1 <- c(1,2,3,1,2,3,1,2,3)
num2 <- c(10,20,30,10,20,30,10,20,30)
num3 <- c(30,20,10,30,20,10,30,20,10)
num4 <- c(12,21,3,0,5,7,22,3,100)
num5 <- c(NA,NA,NA,NA,NA,NA,1,2,3)

df <- cbind.data.frame(cat1,cat2,cat3,cat4,cat5,num1,num2,num3,num4,num5)
df.num <- cbind.data.frame(num1,num2,num3,num4,num5)

cor.matrix.nats <- (mutinformation(discretize(df, disc="equalwidth")))
cor.matrix.nats <- cbind.data.frame(row.names(cor.matrix.nats),cor.matrix.nats)

cor.matrix.num.Spearman <- cor(df.num, method="spearman", use="pairwise.complete.obs")
cor.matrix.num.Spearman <- cbind.data.frame(row.names(cor.matrix.num.Spearman), cor.matrix.num.Spearman)

To my surprise the mutual information cat1 vs. cat2 is a lot higher than cat1 vs. cat3 and in fact I observe the exact same difference for num1 vs. num2 compared to num1 vs. num3 (where Spearman produces the expected result). This leads me to believe that for some reason infotheo somehow uses alphabetical ordering to give unordered categorical variables a non-existing order.

While I checked the documentation and another great explanation on the interpretation of mutual information I still can't get my head around why order seems to play a role...

If anyone's got an explanation or a suggestion how to work around this, it would be highly appreciated!

Answer 1

Ok, I now understand what's happening. The defaults produce only 2 bins which turns

• cat1 and cat2 into 1 2 2 1 2 2 1 2 2
• cat3 into 2 2 1 2 2 1 2 2 1

which explains the lower mutual information value (it is indeed not related to sort order!). Allowing 3 bins by modifying one line of code like so

cor.matrix.nats <- (mutinformation(discretize(df, disc="equalwidth", nbins=NROW(df)^(1/2))))

bins

• cat1 and cat2 into 1 2 3 1 2 3 1 2 3
• cat3 into 3 2 1 3 2 1 3 2 1

and now mutual information cat1 vs. cat2 = cat1 vs. cat3 = 1.0986123.

Guess I need to think about how to get a robust value for nbins in discretize not knowing in advance how many real bins across how many samples are coming in...