I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:
fmla <- as.formula(paste("ex ~ ", paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= "+"))) > bosque <- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000 , cutoff=c(0.2,0.8),importance=TRUE) > bosque Call: randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12, ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE) Type of random forest: classification Number of trees: 1000 No. of variables tried at each split: 12 OOB estimate of error rate: 15.81% Confusion matrix: No Si class.error No 3999 1 0.00025 Si 758 42 0.94750
As we see, the out of bag error for the level "SI" is 0.94750. However If I use the test set to get a sense of the error, the result is very different:
res.arbol <- predict(bosque,test.fullarbol,type="class") > summary(res.arbol) No Si 7761 43 > table(res.arbol,test.fullarbol$ex) res.arbol No Si No 6937 824 Si 4 39 > prop.table(table(res.arbol,test.fullarbol$ex),1) res.arbol No Si No 0.89382811 0.10617189 Si 0.09302326 0.90697674
Now we see the error rate for the test set in the class "Si" is very low , it's equal to 0.093 and It doesn´t make sense to me.
I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?