What's the potential reason that by combining two feature sets the performance of random forest dropped?

I am building random forests on high dimensional, sparse, and class unbalanced training datasets (around 500 - 5000 examples) using two different feature sets. I did stratified 10-fold cross validation. All features are binary, as well as the class label.

Since the training data is class unbalanced, simply voting for majority gives accuracy of 80%. Random forest with feature set 1 (more than 1000 features) gives accuracy of 97%, and random forest with feature set 2 (more than 300 features) gives accuracy of 85%. However, when I merge the two feature sets into one, and then train random forest on the merged feature set, the resulted accuracy is around 95%.

Since using only feature set 2 is better than majority voting, I can consider features in set 2 as useful features. I thought if I add more "useful" features to random forest, the model performance should increase, i.e., I expected the merged feature set to lead to an improved accuracy compared to using only feature set 1.

Due to the class unbalance, I also tired to calculate AUC, however I observed the same thing that using merged features does not outperform using only feature set 1.

What could be the reason for the dropped performance by including more features in random forest? How could I improve the performance of random forest using the merged feature set? Any hint or suggestion would be very much appreciated!

• I suspect that you would need more trees and/or deeper trees if you use more features. – George Dec 1 '14 at 14:52
• @George I am currently using 500 trees as default in randomForest package in R. Do you mean that by building deeper trees will make better use of the additional features? – Jing Zhao Dec 1 '14 at 14:56
• Intuitively, I think that if you add more features you may need trees or even deeper trees to fully "utilize" them. I'm not very experienced with random forests so I may be wrong, but I think it is worth a try. – George Dec 1 '14 at 14:58
• First, try to double the number of trees. You compare two 500-tree RFs with a 500-tree RF and more features. If more features underperforms the better, this is indicative that the RF has too few trees or not deep enough trees. Computation-wise I'd first try to do ntree = 1000. Such a RF has the same number of trees and the same features as a stacking model of the two 500-tree RFs, so it should at least have the potential to outperform it, all else equal. – AlexR Apr 26 '17 at 10:35