I've been trying to improve the performance of my random forest model, and read the following paper on feature selection using random forest (see algorithm in section IV: Overfitting - A. Feature Selection):
http://ftp.cs.nyu.edu/mishra/PUBLICATIONS/Heritage11.pdf
My understanding is, suppose there are 5 predictors: [A, B, C, D, E], the algorithm does the following:
- run_random_forest(data=[A, B, C, D, E], max_features=5) => OOB=0.5, least_important_feature = [B]
- delete [B] from the data file
- run_random_forest(data=[A, C, D, E], max_features=4) => OOB=0.6, least_important_feature = [C]
- delete [C] from the data file
- run_random_forest(data=[A, D, E], max_features=3) => OOB=0.5, least_important_feature = [A]
- Since OOB score in step 5 is smaller than OOB score in step 3, the "optimal" max_features is 4
- run_random_forest(data=[A, B, C, D, E], max_features=4), and rank the feature importance.
Here I have 2 questions:
1) Am I understanding the algorithm correctly?
2) What happens after step 7? If the rank of feature importance after step 7 is D>E>C>B>A with max_features=4, do we then:
- delete feature [A] forever from the data file, and only train the random forest with run_random_forest(data=[B, C, D, E], max_features=4), and predict with [B, C, D, E]?
- or do we still keep feature [A] from the data file, and train the random forest with run_random_forest(data=[A, B, C, D, E], max_feature=4), and predict with [A, B, C, D, E]? Help is really appreciated.
Thanks a lot in advance!
Best Regards,
mangoengineer