How many trees would you suggest to pick to perform recursive feature elimination (RFE) in order to optimize Random Forest classifier (for binary classification problem). My dataset is very high-dimensional (> 200 000 features), and I usually select ~ 10 000 trees while running a classification without feature selection. But I am just wondering whether it is enough to set it as ~ 500-1000 for RFE in order to save time and RAM.

P.S.: I use 'randomForest' and 'caret' R-packages if it makes any difference.


migrated from stackoverflow.com Oct 24 '12 at 12:02

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Breinman's recommendations are to pick a large number of trees, as well as the square root of the number of variables for m. See cwiki.apache.org/MAHOUT/random-forests.html and springerlink.com/index/U0P06167N6173512.pdf $\endgroup$ – lf.xiao Oct 24 '12 at 7:08
  • $\begingroup$ This is something I know. But on the other hand, there is a suggestion that mtry parameter can be adjusted using lower number of trees. My question is wheter it is possible to use lower number of trees for feature selection via RFE. Or do you suggest using the same values? $\endgroup$ – sharky Oct 24 '12 at 7:45

Optimizing ntree and mtry (above mtry=sqrt(#features) and ntree large enough for stabilization of OOB) is a dangerous area -- you need hard core nested cross-validation to be safe from overfitting, so you may end up doing more computations that you are trying to avoid.

I would say the better idea is not to use RFE -- with 200k features it will have terrible requirements and a minor chance to be stable. Instead, you can use some all-relevant RF wrapper like ACE or Boruta -- the set returned is likely to be larger than minimal-optimal, but still way smaller then the original and thus easier to be treated with RFE.

And remember to validate the feature selection regardless of the method (=

  • $\begingroup$ Thanks @mbq. Got it. I was just thinking about performing "rough" RFE (using for example 100%, 90%, ..., 50% of features) with 10-fold cross-validation (CV) prior the 'mtry' adjustment. I just thought that as long as the elimination criteria are based on Gini-index, this should be relatively unbiased given appropriate number of trees and CV. But I see your point... Thanks for the suggestion! $\endgroup$ – sharky Oct 25 '12 at 8:27

I'm actually doing an experiment with this right now. I work in text classification, so my training set is typically on the order of several hundred thousand features, and I'm looking at comparing a linear SVM (optimized for the c-parameter) against the weka implementation of random forests. I'm finding that, for my data, about 74 trees, and 32 features, thus far, seems to give pretty good performance. Of course, increasing these values tends to increase the AUC I observe, but it's in the thousandths digit place, generally. I'm still trying to understand how this algorithm is handling my data, but I suspect, based on the Breiman paper, that the more generally-useful features there are in your training set, the less important the number of trees parameter becomes. If you read the paper (and it's a really great paper), each tree consists of a random sampling of the features in your data, so, if there are a great many helpful ones in your set, you're more likely to find something useful in any particular tree. That said, I think it's always a good idea to optimize an algorithm for one's particular data. For my experiments, I've set aside a training/optimization set on which I am performing cross-validation across different parameter values. I'd be interested to hear what you find!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.