Random forest: overfitting even OOB error is low? Is there any case that OOB ( out of bag) error fails to indicate overfitting? For example OOB is still good but the RF is overfitted.
More specifically,I got low OOB error (8%) with a data set with a lot of wrong labels (i.e. Two samples with very identical feature values may be in different classes and vice versa). The wrong label rate is around 20% of the data set of 7000 samples. The OOB was calculated during training and thus it was based on the wrong labels as well. One possibility may be that the RF was able to learn a very nonlinear cut between two wrong classes even the two classes have a significant overlap. But I want to know if there are any other posibilities.
Thank you.
 A: Elaborating on what @MichaelM said in the comments:
I know of one fairly common situation which in which the OOB error can be extremely misleading. This is when there are many duplicate rows in your training data. If one of the duplicates is in the bag, and the other is out, then the one which is out is very likely to be predicted correctly, which makes the model look better than it is.
I recently ran into this problem when evaluating a model which someone had built after up-sampling the data using SMOTE. The OOB error looked fantastic, but the model was almost useless for classifying unseen data.
Here's a simple example in R of the kind of thing that can happen.
# completely random data
dat <- data.frame(x1=rnorm(1000), x2=rnorm(1000), y=factor(sample(0:1, 1000, replace=T)))
require(randomForest)

# build model after "over-sampling"
dat2 <- rbind(dat, dat)
model <- randomForest(y ~., data=dat2)
model

A: Normally the OOB-Error should not be prone to overfitting, as prediction for each observation is calculated with trees, that have not seen the observation. It is a normal evaluation strategy, like 10 fold-CV.
Only if you have used the OOB before for training your model, e.g. for tuning your hyperparameters, there would be a overfit. 
Your usecase is not so clear to me. Is your outcome binary, or how many classes? How is the proportion of classes? Which classes are wrongly labeled, are they wrongly labeled into the majority class? Because if RF does not know where to classify an observation it probably classifys it into the majority class...
