I am running a random forest classifier using a very large matrix: there are 20,000 incidents and 45,000 features. All of the features are numerical and very sparse. The input matrix thus looks something like this:
Feature_1 Feature_2 Feature_3 Feature_4 ...etc. Incident 1 5 0 1 0 Incident 2 8 3 0 0 Incident 3 0 0 0 0 . . .
Based on ROC analysis, I have determined that of these 45,000 features, about 15,000 can be used to achieve a performance of a 60% True Positive Rate (TPR) at a False Positive Rate (FPR) threshold of 5%. The rest of the 30,000 features combined can only achieve a 20% TPR at a 5% FPR threshold. However, when I combine all of the features together (i.e. training on all 45,000 features), the final model has only a 25% TPR at the same threshold. Can anybody explain why this is happening?
My understanding is that Random Forest is robust to adding uninformative features and will discard less useful features, so that only useful features will be used. To the extent that the other features are useful, they should increase performance at least a little bit. I do not want to do the obvious solution and remove the less informative features because they have other properties that I am interested in.
I am using the package randomForest in R using 50 trees. The Gold Standard is quite small, with only 200 incidents. To create a balanced classifier, I sample 100 incidents of the gold standard and 100 of the negative set, creating a smaller matrix of 200 rows by 45,000 columns, and train on this set and test on the larger matrix (excluding the samples used to train the random forests classifier.) Partially cross-posted from here, albeit with a different question: https://stackoverflow.com/questions/43418776/performance-difference-between-imputation-methods-for-random-forest-classifier