I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable and are almost balanced ] from data and the remaining are result of one hot encoding of a column [ all variables only have two classes ].
So, during feature selection I have performed chi-square test of dependence on my all those variables (and everything as said by the article A Gentle Introduction to the Chi-Squared Test for Machine Learning), with hypothesis:
H0: variables are independent on each other,
H1: variables are dependent on each other.
from scipy import chi2_contingency calculated_statistic, p_value, degrees_of_freedom, expected_values = chi2_contingency(contingency_table)
That's how I generated p-values for all variables with every remaining variable.
Every variable is dependent on minimum of five other variables (p-value < 0.5).
So, my question now is do I discard all those variables.
Here is the data. One-Hot encoded column is named
amenities and all other categorical variables are included as they are, for testing.