I'm currently implementing scikit-learn's RandomForestRegressor in Python and am scratching my head over why I have occasionally wound up with negative out-of-bag scores from it. As far as I can tell from the given description of the attribute "_oob_score" ("Score of the training dataset obtained using an out-of-bag estimate") and everything I've read so far, the out-of-bag score should be a positive value.
All of the scores I have been getting, both positive and negative, are very small in magnitude (< 0.001; many are < 0.0001). I am also not sure if this is normal, but in general the responses are also small.
I'm using 500 trees, and varying min_samples_leaf and max_features. I seem to get the negative values when min_samples_leaf is over ~500.
There are about a hundred Boolean columns that were created to deal with categorical data (that are therefore fairly sparsely populated). In contrast, there are about 10 other, numerical columns. Null values have been filled in with a large negative number as a numerical placeholder.
My data size is about 1,000,000 rows, with 65% being used for training data and the remainder for testing.
(Any other info I can give to help out?)
Is there a statistical interpretation/definition of the out-of-bag score for a random forest for which one would expect a negative score as a possibility, or is this more likely to be a quirk of the program?