0
$\begingroup$

I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable and are almost balanced ] from data and the remaining are result of one hot encoding of a column [ all variables only have two classes ].

So, during feature selection I have performed chi-square test of dependence on my all those variables (and everything as said by the article A Gentle Introduction to the Chi-Squared Test for Machine Learning), with hypothesis:

H0: variables are independent on each other,
H1: variables are dependent on each other.

from scipy import chi2_contingency
calculated_statistic, p_value, degrees_of_freedom, expected_values = chi2_contingency(contingency_table)

That's how I generated p-values for all variables with every remaining variable. Every variable is dependent on minimum of five other variables (p-value < 0.5).
So, my question now is do I discard all those variables.

Here is the data. One-Hot encoded column is named amenities and all other categorical variables are included as they are, for testing.

$\endgroup$
6
  • $\begingroup$ What is your goal? Is your goal to have a list of variables where you are not entirely sure whether they are related? Is your goal to then predict something with this data?In what you are trying to achieve, what do you wish to optimize (e.g. predict something, but it is more important to have a low number of relatively indpeendent predictors - even if predictive accuracy suffers as a result)? $\endgroup$
    – Björn
    Jun 17, 2019 at 20:00
  • $\begingroup$ Keep in mind that with enough data, you can find significant association of arbitrarily small effect size, so you might find statistically significant association between two variables that are only weakly related, in which case you probably don't want to discard any of them. The p-value alone won't tell you if two variables are so similar that you only need one of them. Also, if you're planning on training a classifier downstream, I hope you have already done your train/test split - if you're selecting features from the test data, your performance estimates will be overoptimistic. $\endgroup$ Jun 17, 2019 at 20:02
  • $\begingroup$ @Björn my goal is to predict price of hotels. and to have predictors that are not dependent on each other. $\endgroup$ Jun 18, 2019 at 6:13
  • $\begingroup$ NuclearWang 1. so, how do i decide that the predictors i have are not dependent on each other. 2. I am currently pre-processing the data and haven't made train-test split yet. $\endgroup$ Jun 18, 2019 at 6:20
  • $\begingroup$ The approach I've planned to take is, first clean the data. the data i've obtained from last step will be split into train and test, then select only appropriate features. and then train the model with train data and test the model with test data(with only appropriate features resulting from feature selection) and evaluate model. $\endgroup$ Jun 18, 2019 at 7:51

1 Answer 1

1
$\begingroup$

You say the goal is to predict price of hotels. Why not start out with that, maybe with regularized regression? Why do feature selection at all? See Variable selection for predictive modeling really needed in 2016? and What kind of feature selection can Chi square test be used for?.

So, go for regularized regression followed by validation. Maybe some ideas in Principled way of collapsing categorical variables with many levels? will be useful.

You say my goal is to predict price of hotels. and to have predictors that are not dependent on each other. Why do you want independent features? Concentrate on your final goal!

$\endgroup$
1
  • $\begingroup$ Question: "Why do you want independent features?". Answer: I am doing feature selection and want to take only the variables that really contribute in predicting the target, not the ones that have redundant information. $\endgroup$ Jul 26, 2019 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.