I'm currently implementing scikit-learn's RandomForestRegressor in Python and am scratching my head over why I have occasionally wound up with negative out-of-bag scores from it. As far as I can tell from the given description of the attribute "_oob_score" ("Score of the training dataset obtained using an out-of-bag estimate") and everything I've read so far, the out-of-bag score should be a positive value.

Extra info:

  • All of the scores I have been getting, both positive and negative, are very small in magnitude (< 0.001; many are < 0.0001). I am also not sure if this is normal, but in general the responses are also small.

  • I'm using 500 trees, and varying min_samples_leaf and max_features. I seem to get the negative values when min_samples_leaf is over ~500.

  • There are about a hundred Boolean columns that were created to deal with categorical data (that are therefore fairly sparsely populated). In contrast, there are about 10 other, numerical columns. Null values have been filled in with a large negative number as a numerical placeholder.

  • My data size is about 1,000,000 rows, with 65% being used for training data and the remainder for testing.

  • (Any other info I can give to help out?)

Is there a statistical interpretation/definition of the out-of-bag score for a random forest for which one would expect a negative score as a possibility, or is this more likely to be a quirk of the program?

  • 1
    It's hard for someone who doesn't use this software to be clear whether you have (1) a general statistical question or (2) a question specific to the software. Which is it? (2) would usually be off-topic here. – Nick Cox Jan 14 '15 at 18:37
  • Thanks for the help, @NickCox! I guess the problem is that I don't know (or, that is what I am trying to find out) -- are there any statistical interpretations of out-of-bag scores of a random forest that would allow for a negative score? If not, I am guessing my issue is more in the realm of (2) and I will ask elsewhere, but I suppose I am hoping it is (1) and there is simply something I'm not understanding about the meaning of the term or how it might be applied here. I will edit my question a bit to clarify. – Amanda Jan 14 '15 at 18:58
up vote 5 down vote accepted

RandomForestRegressor's oob_score_ attribute is the score of out-of-bag samples. scikit-learn uses "score" to mean something like "measure of how good a model is", which is different for different models. For RandomForestRegressor (as for most regression models), it's the coefficient of determination, as can be seen by the doc for the score() method.

This is defined as $(1 - u/v)$, where $u$ is the regression's sum squared error $u = \sum_i (y_i - \hat{y}_i)^2$, and $v$ is the sum squared error of the best constant predictor $v = \sum_i (y_i - \bar{y})^2$ (where sums range over the test instances).

This measure can indeed be negative, if $u > v$, i.e. your model is worse than the best constant predictor. This means your model kind of sucks; usually models get positive scores. The score of .0001 or whatever means that your model is only just barely better than the best constant predictor.

  • Thanks so much for the help; the meaning of the score is much clearer to me now. Given the very skewed nature of the data in question and now knowing the oob_score_ is $R^2$, I suppose it's not really a surprise that the score would be that low. We have been able to get reasonable AUCs with the model, though. I wonder if it's justifiable to give more weight to the AUC when deciding whether the model is good, or if the bad $R^2$ indicates a deeper problem that the AUC masks that still needs to be addressed. – Amanda Jan 20 '15 at 15:32

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.