0
$\begingroup$

I am working with a feature selection problem. What I am trying to do is find optimal subset of features for classification.

My data consist of 100 features and 300 instances, and class label is binary. My aim is to detect best feature set that is most correlated to the class label.

First I applied Recursive Feature Elimination(RFE) by Guyon et al. It calculate the weight vectors of all features using SVM, and remove one feature of least weight. It repeats until final subset containing only 1 feature is tested. In this way, all ranking of feature can be acquired.

But it does not suggest guideline for how many features should be selected, at least as I understood.

To find the optimal(Though it is heuristic) subset, I first tried 10-fold cross-validation test and calcuated mean f1 score, removing features one by one backward along the ranking derived from RFE. Namely, 10-fold cv with 100 features, 99 features, ... 1 feature.

However, the result is not uniform. Sometimes 10-feature subset is best, and other times 50-feature subset is best. I think it is due to the lack of test cases, thus I am thinking of testing with random sampling or bagging or repeated cross-validation or something.

This is why I asked previous question about difference between cross-validation and sampling.

In this case, what would be the most appropriate model for the test? My goal is to get stable result(feature set). Also, is there any recommended classifier for this kind of work?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You could use the rfe() function of the caret R package, with SVM as classifier method. With this function, you could choose the resampling method. The function proposes the optimal number of predictors, averaged the results of the resampling. The method is based in the RFE method of Guyon, but incoporates the importance of the variables in each subsample of the resampling loop

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for recommendation. But what I want to know is which resampling method is desirable in my case, becasue one 10-fold cross-validation did not give me a stable result. $\endgroup$
    – ToBeSpecific
    Commented Jan 12, 2016 at 9:29
  • $\begingroup$ The rfe function of caret provides estimation of the measures performance and their standard errors. This could help you to choose a stable resampling method, testing severals of them. $\endgroup$
    – Jesus Herranz Valera
    Commented Jan 12, 2016 at 9:48
  • $\begingroup$ rfe() function seems to be quite convenient to use, at least for testing step. Thank you for your recommendation. $\endgroup$
    – ToBeSpecific
    Commented Jan 12, 2016 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.