sklearn: Is it possible to implement model metrics on Random Forest without creating a separate test set?

Question

I have created a random forest for binary classification. I have a very unbalanced dataset (taken from the actual distribution of the data the model will predict) - tens of thousands of negative cases and only a few hundred positive ones (98% negative). Given the small size of the positive corpus, I would like to refrain as much as possible from creating a separate testing/validation data set.

I believe random forests have a built-in out-of-bag scoring metric (it is also a callable attribute of sklearn random forests). Due to this bagging, the model is already not seeing ~1/3 of the training set (if I am understanding the oob method). Can I somehow leverage this unseen portion of the training set from within sklearn to generate the needed parameters?

The sklearn.metrics functions require y_true: an array of correct labels for some arbitrary list of x's, and y_pred: an array of the labels the model actually gave those arbitrary x's. This would be fairly straightforward to obtain if you have a test set of x's. I want to use the values that were not bagged to construct my test set. How can I discover which training examples were not selected in the bagging process?

Thanks.

Answer 1

The OOB sample is different for every tree in the forest. I don't think the main reason the OOB sample exists is so that it can be used as a test set (although that is certainly a benefit of its existence) the main reason is that it is created as a byproduct of bootstrapping the data for the reason of creating variation in the trees in the forest.

If you are having issues with unbalanced data maybe you could instead try oversampling your data which would make your training set more balanced. I don't know if sklearn allows this but randomForest in R does.

Looking at the randomForest in R documentation again I don't think it oversamples per se, but it does have a classwt variable that you can set to put more reliance on minimizing the error rate for certain classes.