Number of trees for Random Forest optimization using recursive feature elimination

Question

How many trees would you suggest to pick to perform recursive feature elimination (RFE) in order to optimize Random Forest classifier (for binary classification problem). My dataset is very high-dimensional (> 200 000 features), and I usually select ~ 10 000 trees while running a classification without feature selection. But I am just wondering whether it is enough to set it as ~ 500-1000 for RFE in order to save time and RAM.

P.S.: I use 'randomForest' and 'caret' R-packages if it makes any difference.

Answer 1

Optimizing ntree and mtry (above mtry=sqrt(#features) and ntree large enough for stabilization of OOB) is a dangerous area -- you need hard core nested cross-validation to be safe from overfitting, so you may end up doing more computations that you are trying to avoid.

I would say the better idea is not to use RFE -- with 200k features it will have terrible requirements and a minor chance to be stable. Instead, you can use some all-relevant RF wrapper like ACE or Boruta -- the set returned is likely to be larger than minimal-optimal, but still way smaller then the original and thus easier to be treated with RFE.

And remember to validate the feature selection regardless of the method (=

Answer 2

I'm actually doing an experiment with this right now. I work in text classification, so my training set is typically on the order of several hundred thousand features, and I'm looking at comparing a linear SVM (optimized for the c-parameter) against the weka implementation of random forests. I'm finding that, for my data, about 74 trees, and 32 features, thus far, seems to give pretty good performance. Of course, increasing these values tends to increase the AUC I observe, but it's in the thousandths digit place, generally. I'm still trying to understand how this algorithm is handling my data, but I suspect, based on the Breiman paper, that the more generally-useful features there are in your training set, the less important the number of trees parameter becomes. If you read the paper (and it's a really great paper), each tree consists of a random sampling of the features in your data, so, if there are a great many helpful ones in your set, you're more likely to find something useful in any particular tree. That said, I think it's always a good idea to optimize an algorithm for one's particular data. For my experiments, I've set aside a training/optimization set on which I am performing cross-validation across different parameter values. I'd be interested to hear what you find!