I have created a random forest for binary classification. I have a very unbalanced dataset (taken from the actual distribution of the data the model will predict) - tens of thousands of negative cases and only a few hundred positive ones (98% negative). Given the small size of the positive corpus, I would like to refrain as much as possible from creating a separate testing/validation data set.
I believe random forests have a built-in out-of-bag scoring metric (it is also a callable attribute of sklearn random forests). Due to this bagging, the model is already not seeing ~1/3 of the training set (if I am understanding the oob method). Can I somehow leverage this unseen portion of the training set from within sklearn to generate the needed parameters?
The sklearn.metrics functions require y_true: an array of correct labels for some arbitrary list of x's, and y_pred: an array of the labels the model actually gave those arbitrary x's. This would be fairly straightforward to obtain if you have a test set of x's. I want to use the values that were not bagged to construct my test set. How can I discover which training examples were not selected in the bagging process?