What is a good oob score for random forests with sklearn, three-class classification? [duplicate]

Question

I have learning data consisting of ~45k samples, each has 21 features. I am trying to train a random forest classifier on this data, which is labelled to 3 classes (-1, 0 and 1). The classes are more or less equal in their sizes.

My random forest classifier model is using gini as its split quality criterion, the number of trees is 10, and I have not limited the depth of a tree.

Most of the features have shown negligible importance - the mean is about 5%, a third of them is of importance 0, a third of them is of importance above the mean.

However, perhaps the most striking fact is the oob (out-of-bag) score: a bit less than 1%. It made me think the model fails, and indeed, testing the model on a new independent set of size ~40k, I got score of 63% (sounds good so far), but a deeper inspection of the confusion matrix have shown me that the model only succeeds for class 0, and fails in about 50% of the cases when it comes to decide between 1 and -1.

Python's output attached:

array([[ 7732,   185,  6259],
       [  390, 11506,   256],
       [ 7442,   161,  6378]])

This is naturally because the 0 class has special properties which makes it much easier to predict. However, is it true that the oob score I've found is already a sign that the model is not good? What is a good oob score for random forests? Is there some law-of-thumb which helps determining whether a model is "good", using the oob score alone, or in combination with some other results of the model?

---

Edit: after removing bad data (about third of the data), the labels were more or less 2% for 0 and 49% for each of -1/+1. The oob score was 0.011 and the score on the test data was 0.49, with confusion matrix hardly biased towards class 1 (about 3/4 of the predictions).

Answer 1

sklearn's RF oob_score_ (note the trailing underscore) seriously isn't very intelligible compared to R's, after reading the sklearn doc and source code. My advice on how to improve your model is as follows:

1. sklearn's RF used to use the terrible default of max_features=1 (as in "try every feature on every node"). Then it's no longer doing random column(/feature)-selection like a random-forest. Change this to e.g.max_features=0.33 (like R's mtry) and rerun. Tell us the new scores.

2. "Most of the features have shown negligible importance". Then you need to do Feature Selection, as per the doc - for classification. See the doc and other articles here on CrossValidated.SE. Do the FS on a different (say 20-30%) holdout set than the rest of the training, using e.g. sklearn.cross_validation.train_test_split() (yes the name is a bit misleading). Now tell us the scores you get after FS?

3. You said "after removing bad data (about third of the data), the labels were more or less 2% for 0 and 49% for each of -1/+1" ; then you have a severe class imbalance. Also: "confusion matrix shows model only succeeds for class 0, and fails in about 50% of the cases between +1 and -1". This is a symptom of the class imbalance. Either you use stratified sampling, or train a classifier with examples for +1 and -1 class. You can either do a OAA (One-Against-All) or OAO (One-Against-One) classifier. Try three OAA classifiers, one for each class. Finally tell us those scores?

Answer 2

There's no such thing as good oob_score, its the difference between valid_score and oob_score that matters.

Think of oob_score as a score for some subset(say, oob_set) of training set. To learn how its created refer this.

oob_set is taken from your training set. And you already have your validation set(say, valid_set).

Lets assume a scenario where, your validation_score is 0.7365 and oob_score is 0.8329

In this scenario, your model is performing better on oob_set, which is take directly from your training dataset. Indicating, validation_set is for a different time period. (say training_set has records for the month of "January" and validation_set has records for the month of "July"). So, more than a test for model's performance, oob_score is test for "how representative is your Validation_set".

You should always make sure that you have a good representative validation_set, because it's score is used as an indicator for our model's performance. So your goal should be, to have as little difference between oob_score and valid_score as possible.

I generally use oob_score with validation_score to see how good is my validation_set. I learnt this technique from Jeremy Howard.

Answer 3

Q: What is a good oob score for random forests with sklearn, three-class classification?

A: Depends. In my view, if learning and testing samples are drawn from the same distribution, then -in my view- OOB is equal to approximately 3-fold cross-validation. So if we repeat the same question but with "3-fold cross-validation", the answer would be the same, which is "generally, the highest the accuracy the merrier, unless you fear to overfit your learning set because someone told you that the true testing samples are of a different distribution".

Can you give me your dataset? I can have little fun with it and tell you what I manage to do with it for free.

Answer 4

a different take on the question: to start with, you have to associate a loss with every misclassification you do. This price-paid/loss/penalty for misclassification would(probably) be different for False Positive(FP) vs False Negatives(FN). Some classifications, say cancer detection, would rather have more FPs than FNs. Some other, say spam filter, would rather allow certain spams(FN) than block mails(FP) from your friend. Building on this logic you can have use F1-score or Accuracy, whatever suits your purpose.( for eg. I could be happy if my spam filter has no FPs and a score of .1 as I have 10% less spams to worry about. On the other hand someone else could be unhappy with even .9 (90% spams filtered). What would be good score then?)