I have dataset with different types of features (numeric/continuous and binary). Altogether 66 features.

I want to find a small subset of features that is sufficient for a multi class classification problems (3 classes).

Therefore, I would like to do a filter-based feature selection using a technique that gives a p-value per feature so that I can take the best 5 features based on their p-values, for example.

For the numeric features I can use the ANOVA test to get the p-values and for the binary features I could use $\chi^2$ test to get the p-values.

Now I have the following two questions:

  1. Can I compare the p-values computed by ANOVA and the p-values computed by $\chi^2$ to rank all features based on the p-values?
  2. Is it okay to use ANOVA on the binary features (the binary features are treated as if they were continuous) and compare the p-values to the p-values of numeric/continuous features?

What you apparently want to do is to start by evaluating the relationship between the 3 classes and each of your features individually. For each continuous feature you are proposing a one-way ANOVA of that feature against the known classification (3 classes). ANOVA is not appropriate for a similar evaluation of the binary features, as interpretation of ANOVA p-values assumes that residuals about the mean values are normally distributed. That can't be the case for your binary features. So the answer to your second question is "no."

With respect to your first question, your one-at-a-time approach has a big potential problem: it could miss features that are important for classification when other features are taken into account. So don't try to rank-order the p-values determined on individual features. You would be better off with an approach that includes as much information as possible about all of your features to start with. Depending on the size of your data set, it might be possible to include all features directly while avoiding overfitting. Otherwise you could use your knowledge of the subject matter to exclude or combine some features, or use a penalized approach like ridge regression or LASSO.

  • $\begingroup$ Various sources say that ANOVA is robust to various violations of assumptions. For binary features both assumptions 1) normality of residuals and 2) equal variance are violated, but my experiments with real data show that ANOVA works well when applied to binary features. Using ANOVA I found a subset containing 6 features (of the 66 features) with 3 continuous and 3 binary features. With this subset I can achieve the same accuracy results as with all features. Would this argumentation be sufficient to continue using ANOVA? $\endgroup$ – methus Feb 15 at 12:51
  • 1
    $\begingroup$ @methus a linear probability model as with ANOVA can work pretty well for point estimates provided probabilities are in the middle range (say, 0.2 to 0.8); see this page for example. P-values would not be correct in terms of actual false-positive rates; for rank-ordering individual features it might work OK. But it won't work so well if some classes have very low or high probabilities, and the one-feature-at-a-time approach has dangers noted in the answer. Going forward, why not use methods that apply to wider classes of situations? $\endgroup$ – EdM Feb 15 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.