# Strange difference between Out of bag error and test Error in Random Forest

I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:

fmla <- as.formula(paste("ex ~ ", paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= "+")))
> bosque <- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000  , cutoff=c(0.2,0.8),importance=TRUE)
> bosque

Call:
randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12,      ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE)
Type of random forest: classification
Number of trees: 1000
No. of variables tried at each split: 12

OOB estimate of  error rate: 15.81%
Confusion matrix:
No Si class.error
No 3999  1     0.00025
Si  758 42     0.94750


As we see, the out of bag error for the level "SI" is 0.94750. However If I use the test set to get a sense of the error, the result is very different:

res.arbol <- predict(bosque,test.fullarbol,type="class")
> summary(res.arbol)
No   Si
7761   43
> table(res.arbol,test.fullarbol$ex) res.arbol No Si No 6937 824 Si 4 39 > prop.table(table(res.arbol,test.fullarbol$ex),1)

res.arbol         No         Si
No 0.89382811 0.10617189
Si 0.09302326 0.90697674


Now we see the error rate for the test set in the class "Si" is very low , it's equal to 0.093 and It doesn´t make sense to me.

I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?

• It could be test shows a very similar confusion matrix, just that it transposed. Neither the print.randomForest or table confusion matrix states clearly if rows are true class or predicted class. Oct 4, 2015 at 21:53

You have transposed the confusion matrix. See test example.

library(randomForest)
#simulate data set
set.seed(123)
obs = 4000+800+200
vars = 6
noise.factor=.2
X =  data.frame(replicate(vars,rnorm(obs)))
yvalue = X[,1]^2+sin(X[,2]) + rnorm(obs) * noise.factor
y = factor(c("No","Si")[(yvalue<=quantile(yvalue, 800/4800))*1+1])
test = sample(obs,200)
rf = randomForest(X[-test,],y[-test],ntree=50,cutoff=c(.2,.8))
print(rf)

#yep this confusion table is inverted compared to print.randomForest
table(predict(rf,X[test,]),y[test])

#here row's a true class, and columns are predicted class
table(y[test],predict(rf,X[test,],))

#cutoff is inherrited from training, but can also be modified during prediction
table(y[test],predict(rf,X[test,]))
table(y[test],predict(rf,X[test,],cutoff=c(.2,.8)))
table(y[test],predict(rf,X[test,],cutoff=c(.7,.000001)))