Lets say I have 50 features and want to select few relevant features (say around 20 features) for binary (2 class) classification. I have studied that we can use p values to decide which features are more relevant (p-value < 0.05). Please tell me which statistical test (chi-square/t-test/z-test) should I use to find p-value? Also tell me how to use that test? Can you please give me some reference regarding this?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I have no advice on what type of test to use for binary classification, but I do strongly advise you not to dichotomise the P-values into above and below 0.05. Rank the interestingness of features using the exact values of the P-values. $\endgroup$– Michael LewCommented Mar 18, 2016 at 20:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
I do not recommend to use the p-value to select the features. I think what you want to do is regressing the binary variable on the 50 features, then you get ride of the features that does not have a significant p-value. This is a wrong approach. The null hypothesis of the logistic regression model is that all the coefficients of the features are 0. Therefore, this is a simultaneous testing. Getting rid of any feature can result in change of the distribution for the original model and features. Due to the correlation among the variables, you cannot conclude from the small p-value and say the corresponding feature is important, vice versa.
However, using the logistic function, regressing the binary response variable on the 50 features, is a convenient and quick method of taking a quick look at the data and learn the features. Here is a tutorial on how to do a quick logistic regression in R logistic regression UCLA. By doing this, you will at least know which factor might be important, and which might not.
If I were you, I will think about if this data really makes any intuitive senses. If this is a real world problem, your intuition can help you eliminate some of the "impossible" variables.
Then, random forest might be a good approach. It is a boosted tree-based model. It has the importance rank that you can look at. It basically ranks the features by importance. You can do this in R. Here is the tutorial that you can look into random forest Since your response variable is a binary variable, random forest is good for the supervised classification problem.
You can also do a lasso regression. Here is a good forum discussing on this topic that is similar to yours Lasso Example
-
$\begingroup$ Thanks Shijia Bian for answer. I will try your suggested methods and let you know. $\endgroup$ Commented Mar 27, 2016 at 6:52