How does feature selection work in Random Forest?

Question

I've been trying to improve the performance of my random forest model, and read the following paper on feature selection using random forest (see algorithm in section IV: Overfitting - A. Feature Selection):

http://ftp.cs.nyu.edu/mishra/PUBLICATIONS/Heritage11.pdf

My understanding is, suppose there are 5 predictors: [A, B, C, D, E], the algorithm does the following:

1. run_random_forest(data=[A, B, C, D, E], max_features=5) => OOB=0.5, least_important_feature = [B]
2. delete [B] from the data file
3. run_random_forest(data=[A, C, D, E], max_features=4) => OOB=0.6, least_important_feature = [C]
4. delete [C] from the data file
5. run_random_forest(data=[A, D, E], max_features=3) => OOB=0.5, least_important_feature = [A]
6. Since OOB score in step 5 is smaller than OOB score in step 3, the "optimal" max_features is 4
7. run_random_forest(data=[A, B, C, D, E], max_features=4), and rank the feature importance.

Here I have 2 questions:

1) Am I understanding the algorithm correctly?

2) What happens after step 7? If the rank of feature importance after step 7 is D>E>C>B>A with max_features=4, do we then:

• delete feature [A] forever from the data file, and only train the random forest with run_random_forest(data=[B, C, D, E], max_features=4), and predict with [B, C, D, E]?
• or do we still keep feature [A] from the data file, and train the random forest with run_random_forest(data=[A, B, C, D, E], max_feature=4), and predict with [A, B, C, D, E]? Help is really appreciated.

Thanks a lot in advance!

Best Regards,

mangoengineer

Answer 1

Hope this will help (the section 2 of the paper)

"Variable Selection in Random Forest with Application to Quantitative Structure-Activity Relationship", Vladimir Svetnik, Andy Liaw, and Christopher Tong, Biometrics Research