I'm trying to assess the validity of Random Forests models that I'm running in Python with sklearn. I'm testing the same input data with a few different parameters (n_trees and max_features). Even though I know cross-validation is said to be inherent in the random forests models, I'm trying to do this for a simple school project and so am doing traditional cross-validation as well as looking at the oob scores.

I am using the model to identify Water. My two classes are Water and Not Water. The actual classification maps look good and seem to identify water really well. In all cases, the issue is that sklearn's oob_score_ is returning extremely low scores, almost 0, but when I test with the samples set aside for validation, the accuracy is very high for both classes:

model.oob_score_ = 0.000920306768923

Cross tabs:

predict      1      2    All
truth                       
1        30980    293  31273
2          140  26244  26384
All      31120  26537  57657

It's weird to me that the results for regular CV and OOB scores are almost opposite. What could be going on?