0
$\begingroup$

I'm trying to assess the validity of Random Forests models that I'm running in Python with sklearn. I'm testing the same input data with a few different parameters (n_trees and max_features). Even though I know cross-validation is said to be inherent in the random forests models, I'm trying to do this for a simple school project and so am doing traditional cross-validation as well as looking at the oob scores.

I am using the model to identify Water. My two classes are Water and Not Water. The actual classification maps look good and seem to identify water really well. In all cases, the issue is that sklearn's oob_score_ is returning extremely low scores, almost 0, but when I test with the samples set aside for validation, the accuracy is very high for both classes:

model.oob_score_ = 0.000920306768923

Cross tabs:

predict      1      2    All
truth                       
1        30980    293  31273
2          140  26244  26384
All      31120  26537  57657

It's weird to me that the results for regular CV and OOB scores are almost opposite. What could be going on?

Improve this question
$\endgroup$
2
  • $\begingroup$ Did you mean to say CV accuracy is low rather than high? $\endgroup$
    – Michael R. Chernick
    Commented Nov 20, 2017 at 19:19
  • $\begingroup$ @MichaelChernick no, the overall CV accuracy was very high (~99.2%) and the OOB score was almost 0. However it seems to have been a bug with sklearn 0.14. I installed the most recent version and now my OOB score is similar to the CV. $\endgroup$
    – user20408
    Commented Nov 21, 2017 at 15:37

1 Answer 1

Reset to default
1
$\begingroup$

Turns out this was bug with the software, sklearn 0.14. I got the most recent version (0.19) and now my OOB score is very high, like the CV.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.