I've started to study feature selection techniques and i have a situation that I don't understand. I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are as follow:

X1 = random boolean (p=0.5)

X2 = random boolean (p=0.5)

X3 = random boolean (p=0.5)

X4 = X2 XOR X3

X5 = random boolean (p=0.5)

Y = XOR(X1, X2, X3)

So when I apply mutual information criteria to rank the relevance of the variables I'm getting that X5 is the most relevant, as far as I know it should be the lower in the rank because Y does not depend at all of it.

Here is my code:

import numpy as np
from sklearn.feature_selection import mutual_info_classif as MIC
from Py_FS.filter import Relief

SAMPLE_SIZE = 100

def xor_three(a, b, c):
    return (~a*~b*c)+(~a*b*~c)+(a*~b*~c)+(a*b*c)

def xor_two(a, b):
    return a != b

X1 = np.random.choice([False, True], size = SAMPLE_SIZE)
X2 = np.random.choice([False, True], size = SAMPLE_SIZE)
X3 = np.random.choice([False, True], size = SAMPLE_SIZE)
X4 = xor_two(X2, X3)
X5 = np.random.choice([False, True], size = SAMPLE_SIZE)
X = np.column_stack([X1,X2,X3,X4,X5])
Y  = xor_three(X1, X2, X3)

mi_score = MIC(X,Y, discrete_features=True)
print(mi_score)

As an example in one of the execution I've got in mi_score -> [0.00163452 0.00068499 0.00501039 0.01996895 0.02920959]

Not sure if I've missundertood the concept or I am doing something bad.

I've started to study feature selection techniques and i have a situation that I don't understand. I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are as follow:

X1 = random boolean (p=0.5)

X2 = random boolean (p=0.5)

X3 = random boolean (p=0.5)

X4 = XOR(X2, X3)

X5 = random boolean (p=0.5)

Y = XOR(X1, X2, X3)

So when I apply mutual information criteria to rank the relevance of the variables I'm getting that X5 is the most relevant, as far as I know it should be the lower in the rank because Y does not depend at all of it.

Here is my code:

import numpy as np
from sklearn.feature_selection import mutual_info_classif as MIC
from Py_FS.filter import Relief

SAMPLE_SIZE = 100

def xor_three(a, b, c):
    return (~a*~b*c)+(~a*b*~c)+(a*~b*~c)+(a*b*c)

def xor_two(a, b):
    return a != b

X1 = np.random.choice([False, True], size = SAMPLE_SIZE)
X2 = np.random.choice([False, True], size = SAMPLE_SIZE)
X3 = np.random.choice([False, True], size = SAMPLE_SIZE)
X4 = xor_two(X2, X3)
X5 = np.random.choice([False, True], size = SAMPLE_SIZE)
X = np.column_stack([X1,X2,X3,X4,X5])
Y  = xor_three(X1, X2, X3)

mi_score = MIC(X,Y, discrete_features=True)
print(mi_score)

As an example in one of the execution I've got in mi_score -> [0.00163452 0.00068499 0.00501039 0.01996895 0.02920959]

Not sure if I've missundertood the concept or I am doing something bad.

I've started to study feature selection techniques and i have a situation that I don't understand. I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are as follow: X1 = random boolean (p=0.5)

X2 = random boolean (p=0.5)

X3 = random boolean (p=0.5)

X4 = X2 XOR X3

X5 = random boolean (p=0.5)

Y = XOR(X1, X2, X3)

So when I apply mutual information criteria to rank the relevance of the variables I'm getting that X5 is the most relevant, as far as I know it should be the lower in the rank because Y does not depend at all of it.

Here is my code:

import numpy as np
from sklearn.feature_selection import mutual_info_classif as MIC
from Py_FS.filter import Relief

SAMPLE_SIZE = 100

def xor_three(a, b, c):
    return (~a*~b*c)+(~a*b*~c)+(a*~b*~c)+(a*b*c)

def xor_two(a, b):
    return a != b

X1 = np.random.choice([False, True], size = SAMPLE_SIZE)
X2 = np.random.choice([False, True], size = SAMPLE_SIZE)
X3 = np.random.choice([False, True], size = SAMPLE_SIZE)
X4 = xor_two(X2, X3)
X5 = np.random.choice([False, True], size = SAMPLE_SIZE)
X = np.column_stack([X1,X2,X3,X4,X5])
Y  = xor_three(X1, X2, X3)

mi_score = MIC(X,Y, discrete_features=True)
print(mi_score)

As an example in one of the execution I've got in mi_score -> [0.00163452 0.00068499 0.00501039 0.01996895 0.02920959]

Not sure if I've missundertood the concept or I am doing something bad.

I've started to study feature selection techniques and i have a situation that I don't understand. I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are as follow:

X1 = random boolean (p=0.5)

X2 = random boolean (p=0.5)

X3 = random boolean (p=0.5)

X4 = X2 XOR X3

X5 = random boolean (p=0.5)

Y = XOR(X1, X2, X3)

So when I apply mutual information criteria to rank the relevance of the variables I'm getting that X5 is the most relevant, as far as I know it should be the lower in the rank because Y does not depend at all of it.

Here is my code:

import numpy as np
from sklearn.feature_selection import mutual_info_classif as MIC
from Py_FS.filter import Relief

SAMPLE_SIZE = 100

def xor_three(a, b, c):
    return (~a*~b*c)+(~a*b*~c)+(a*~b*~c)+(a*b*c)

def xor_two(a, b):
    return a != b

X1 = np.random.choice([False, True], size = SAMPLE_SIZE)
X2 = np.random.choice([False, True], size = SAMPLE_SIZE)
X3 = np.random.choice([False, True], size = SAMPLE_SIZE)
X4 = xor_two(X2, X3)
X5 = np.random.choice([False, True], size = SAMPLE_SIZE)
X = np.column_stack([X1,X2,X3,X4,X5])
Y  = xor_three(X1, X2, X3)

mi_score = MIC(X,Y, discrete_features=True)
print(mi_score)

As an example in one of the execution I've got in mi_score -> [0.00163452 0.00068499 0.00501039 0.01996895 0.02920959]

Not sure if I've missundertood the concept or I am doing something bad.