I am using python sklearn.ensemble to do a RandomForestClassifier on about 800K rows of data, coupled with sklearn.cross_validation to generate the train/test sets. When it completes, it says on the test set it achieves 90+% precision and 30+% recall and similar F1-score.
When I manually leave out a continuous chunk of about 10K rows of data, however, running predict() on those rows sucks big time...less than 5% precision and nearly no recall or F1-score.
What gives? Why would the auto-selected test sets (from sklearn.cross_validation) predict so well, but my manually-selected test sets not be predictable at all?
I am generating 20 trees (n_estimators=20) and 8-fold cross-validation (cv=8) and my feature set is about 60.