Mutual Information result in python for Feature selection unexpected result

Question

I've started to study feature selection techniques and i have a situation that I don't understand. I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are as follow:

X1 = random boolean (p=0.5)

X2 = random boolean (p=0.5)

X3 = random boolean (p=0.5)

X4 = XOR(X2, X3)

X5 = random boolean (p=0.5)

Y = XOR(X1, X2, X3)

So when I apply mutual information criteria to rank the relevance of the variables I'm getting that X5 is the most relevant, as far as I know it should be the lower in the rank because Y does not depend at all of it.

Here is my code:

import numpy as np
from sklearn.feature_selection import mutual_info_classif as MIC
from Py_FS.filter import Relief

SAMPLE_SIZE = 100

def xor_three(a, b, c):
    return (~a*~b*c)+(~a*b*~c)+(a*~b*~c)+(a*b*c)

def xor_two(a, b):
    return a != b

X1 = np.random.choice([False, True], size = SAMPLE_SIZE)
X2 = np.random.choice([False, True], size = SAMPLE_SIZE)
X3 = np.random.choice([False, True], size = SAMPLE_SIZE)
X4 = xor_two(X2, X3)
X5 = np.random.choice([False, True], size = SAMPLE_SIZE)
X = np.column_stack([X1,X2,X3,X4,X5])
Y  = xor_three(X1, X2, X3)

mi_score = MIC(X,Y, discrete_features=True)
print(mi_score)

As an example in one of the execution I've got in mi_score -> [0.00163452 0.00068499 0.00501039 0.01996895 0.02920959]

Not sure if I've missundertood the concept or I am doing something bad.

Answer 1

The problem is that none of your predictors are associated to the target (i.e. have a MI value that is not due to basically random noise).
Even though your target is calculated based on the three other predictors the resulting output is completely unrelated to any single variable that was used.
To give an easier example let us consider your X4 as target variable which is calculated based only on X2 and X3:

$$ \begin{array}{c|c|c} X2 & X3 & X4\\ \hline 1 & 1 & 0 \\ 0 & 0 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 0 \\ 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \\ 0 & 1 & 1 \end{array} $$ If you calculate the mutual information between the columns of this table you get the following values:

$$ \begin{equation} MI(X2;X4) = 0\\ MI(X3;X4) = 0 \end{equation} $$

There is no association between input and output. The same would happen with your xor_three function.
That X5 obtains the largest MI value in your example is just due to randomness. You could repeat it and get another variable as top predictor.