Strange difference between Out of bag error and test Error in Random Forest

Question

I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:

fmla <- as.formula(paste("ex ~ ", paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= "+")))
> bosque <- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000  , cutoff=c(0.2,0.8),importance=TRUE)
> bosque

Call:
 randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12,      ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE) 
               Type of random forest: classification
                     Number of trees: 1000
No. of variables tried at each split: 12

        OOB estimate of  error rate: 15.81%
Confusion matrix:
     No Si class.error
No 3999  1     0.00025
Si  758 42     0.94750

As we see, the out of bag error for the level "SI" is 0.94750. However If I use the test set to get a sense of the error, the result is very different:

res.arbol <- predict(bosque,test.fullarbol,type="class")
> summary(res.arbol)
  No   Si 
7761   43 
> table(res.arbol,test.fullarbol$ex)

res.arbol   No   Si
       No 6937  824
       Si    4   39
> prop.table(table(res.arbol,test.fullarbol$ex),1)

res.arbol         No         Si
       No 0.89382811 0.10617189
       Si 0.09302326 0.90697674

Now we see the error rate for the test set in the class "Si" is very low , it's equal to 0.093 and It doesn´t make sense to me.

I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?

Answer 1

You have transposed the confusion matrix. See test example.

library(randomForest)
#simulate data set
set.seed(123)
obs = 4000+800+200
vars = 6
noise.factor=.2
X =  data.frame(replicate(vars,rnorm(obs)))
yvalue = X[,1]^2+sin(X[,2]) + rnorm(obs) * noise.factor
y = factor(c("No","Si")[(yvalue<=quantile(yvalue, 800/4800))*1+1])
test = sample(obs,200)
rf = randomForest(X[-test,],y[-test],ntree=50,cutoff=c(.2,.8))
print(rf)

#yep this confusion table is inverted compared to print.randomForest
table(predict(rf,X[test,]),y[test])

#here row's a true class, and columns are predicted class
table(y[test],predict(rf,X[test,],))

#cutoff is inherrited from training, but can also be modified during prediction
table(y[test],predict(rf,X[test,]))
table(y[test],predict(rf,X[test,],cutoff=c(.2,.8)))
table(y[test],predict(rf,X[test,],cutoff=c(.7,.000001)))