5
$\begingroup$

I've been trying to improve the performance of my random forest model, and read the following paper on feature selection using random forest (see algorithm in section IV: Overfitting - A. Feature Selection):

http://ftp.cs.nyu.edu/mishra/PUBLICATIONS/Heritage11.pdf

My understanding is, suppose there are 5 predictors: [A, B, C, D, E], the algorithm does the following:

  1. run_random_forest(data=[A, B, C, D, E], max_features=5) => OOB=0.5, least_important_feature = [B]
  2. delete [B] from the data file
  3. run_random_forest(data=[A, C, D, E], max_features=4) => OOB=0.6, least_important_feature = [C]
  4. delete [C] from the data file
  5. run_random_forest(data=[A, D, E], max_features=3) => OOB=0.5, least_important_feature = [A]
  6. Since OOB score in step 5 is smaller than OOB score in step 3, the "optimal" max_features is 4
  7. run_random_forest(data=[A, B, C, D, E], max_features=4), and rank the feature importance.

Here I have 2 questions:

1) Am I understanding the algorithm correctly?

2) What happens after step 7? If the rank of feature importance after step 7 is D>E>C>B>A with max_features=4, do we then:

  • delete feature [A] forever from the data file, and only train the random forest with run_random_forest(data=[B, C, D, E], max_features=4), and predict with [B, C, D, E]?
  • or do we still keep feature [A] from the data file, and train the random forest with run_random_forest(data=[A, B, C, D, E], max_feature=4), and predict with [A, B, C, D, E]? Help is really appreciated.

Thanks a lot in advance!

Best Regards,

mangoengineer

$\endgroup$
1
  • $\begingroup$ The link is dead... $\endgroup$
    – user88
    Dec 15, 2015 at 9:17

1 Answer 1

3
$\begingroup$

Hope this will help (the section 2 of the paper)

"Variable Selection in Random Forest with Application to Quantitative Structure-Activity Relationship", Vladimir Svetnik, Andy Liaw, and Christopher Tong, Biometrics Research

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.