I am building random forests on high dimensional, sparse, and class unbalanced training datasets (around 500 - 5000 examples) using two different feature sets. I did stratified 10-fold cross validation. All features are binary, as well as the class label.
Since the training data is class unbalanced, simply voting for majority gives accuracy of 80%. Random forest with feature set 1 (more than 1000 features) gives accuracy of 97%, and random forest with feature set 2 (more than 300 features) gives accuracy of 85%. However, when I merge the two feature sets into one, and then train random forest on the merged feature set, the resulted accuracy is around 95%.
Since using only feature set 2 is better than majority voting, I can consider features in set 2 as useful features. I thought if I add more "useful" features to random forest, the model performance should increase, i.e., I expected the merged feature set to lead to an improved accuracy compared to using only feature set 1.
Due to the class unbalance, I also tired to calculate AUC, however I observed the same thing that using merged features does not outperform using only feature set 1.
What could be the reason for the dropped performance by including more features in random forest? How could I improve the performance of random forest using the merged feature set? Any hint or suggestion would be very much appreciated!