2
$\begingroup$

Is there any case that OOB ( out of bag) error fails to indicate overfitting? For example OOB is still good but the RF is overfitted.

More specifically,I got low OOB error (8%) with a data set with a lot of wrong labels (i.e. Two samples with very identical feature values may be in different classes and vice versa). The wrong label rate is around 20% of the data set of 7000 samples. The OOB was calculated during training and thus it was based on the wrong labels as well. One possibility may be that the RF was able to learn a very nonlinear cut between two wrong classes even the two classes have a significant overlap. But I want to know if there are any other posibilities.

Thank you.

Improve this question
$\endgroup$
3
  • $\begingroup$ Please define your abbreviations, what is OOB? $\endgroup$
    – Matthew Drury
    Jul 14, 2016 at 16:10
  • $\begingroup$ Good question! I am also very interested in how OOB error relates to predictive performance. $\endgroup$
    – Flounderer
    Jul 15, 2016 at 3:38
  • 1
    $\begingroup$ OOB error to be accurate requires the observations to be independent. Is this the case in your setting? $\endgroup$
    – Michael M
    Jul 17, 2016 at 7:19

2 Answers 2

Reset to default
3
$\begingroup$

Elaborating on what @MichaelM said in the comments:

I know of one fairly common situation which in which the OOB error can be extremely misleading. This is when there are many duplicate rows in your training data. If one of the duplicates is in the bag, and the other is out, then the one which is out is very likely to be predicted correctly, which makes the model look better than it is.

I recently ran into this problem when evaluating a model which someone had built after up-sampling the data using SMOTE. The OOB error looked fantastic, but the model was almost useless for classifying unseen data.

Here's a simple example in R of the kind of thing that can happen.

# completely random data
dat <- data.frame(x1=rnorm(1000), x2=rnorm(1000), y=factor(sample(0:1, 1000, replace=T)))
require(randomForest)

# build model after "over-sampling"
dat2 <- rbind(dat, dat)
model <- randomForest(y ~., data=dat2)
model
Improve this answer
$\endgroup$
1
  • $\begingroup$ More generally speaking, if the bootstrapping doesn't take into account any structure in the data, the splits are not independent. Thus, the guard against the known overfitting of the single trees is basically disarmed. $\endgroup$
    – cbeleites unhappy with SX
    Nov 1, 2019 at 1:08
0
$\begingroup$

Normally the OOB-Error should not be prone to overfitting, as prediction for each observation is calculated with trees, that have not seen the observation. It is a normal evaluation strategy, like 10 fold-CV.

Only if you have used the OOB before for training your model, e.g. for tuning your hyperparameters, there would be a overfit.

Your usecase is not so clear to me. Is your outcome binary, or how many classes? How is the proportion of classes? Which classes are wrongly labeled, are they wrongly labeled into the majority class? Because if RF does not know where to classify an observation it probably classifys it into the majority class...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.