I am using python sklearn.ensemble to do a RandomForestClassifier on about 800K rows of data, coupled with sklearn.cross_validation to generate the train/test sets. When it completes, it says on the test set it achieves 90+% precision and 30+% recall and similar F1-score.

When I manually leave out a continuous chunk of about 10K rows of data, however, running predict() on those rows sucks big time...less than 5% precision and nearly no recall or F1-score.

What gives? Why would the auto-selected test sets (from sklearn.cross_validation) predict so well, but my manually-selected test sets not be predictable at all?

I am generating 20 trees (n_estimators=20) and 8-fold cross-validation (cv=8) and my feature set is about 60.

  • $\begingroup$ Hard to be specific without sight of the data, but as you say, you might expect better performance than you observe with the manually reduced set. You say you remove 10K of contiguous data, do you see the same result if you remove 10K of random data ...? Perhaps there is something special about the data you have removed. $\endgroup$ – image_doctor Dec 31 '14 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.