I've been trying to improve the performance of my random forest model, and read the following paper on feature selection using random forest (see algorithm in section IV: Overfitting - A. Feature Selection):

http://ftp.cs.nyu.edu/mishra/PUBLICATIONS/Heritage11.pdf

My understanding is, suppose there are 5 predictors: [A, B, C, D, E], the algorithm does the following:

  1. run_random_forest(data=[A, B, C, D, E], max_features=5) => OOB=0.5, least_important_feature = [B]
  2. delete [B] from the data file
  3. run_random_forest(data=[A, C, D, E], max_features=4) => OOB=0.6, least_important_feature = [C]
  4. delete [C] from the data file
  5. run_random_forest(data=[A, D, E], max_features=3) => OOB=0.5, least_important_feature = [A]
  6. Since OOB score in step 5 is smaller than OOB score in step 3, the "optimal" max_features is 4
  7. run_random_forest(data=[A, B, C, D, E], max_features=4), and rank the feature importance.

Here I have 2 questions:

1) Am I understanding the algorithm correctly?

2) What happens after step 7? If the rank of feature importance after step 7 is D>E>C>B>A with max_features=4, do we then:

  • delete feature [A] forever from the data file, and only train the random forest with run_random_forest(data=[B, C, D, E], max_features=4), and predict with [B, C, D, E]?
  • or do we still keep feature [A] from the data file, and train the random forest with run_random_forest(data=[A, B, C, D, E], max_feature=4), and predict with [A, B, C, D, E]? Help is really appreciated.

Thanks a lot in advance!

Best Regards,

mangoengineer

  • The link is dead... – mbq Dec 15 '15 at 9:17

Hope this will help (the section 2 of the paper)

"Variable Selection in Random Forest with Application to Quantitative Structure-Activity Relationship", Vladimir Svetnik, Andy Liaw, and Christopher Tong, Biometrics Research

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.