1
$\begingroup$

I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:

fmla <- as.formula(paste("ex ~ ", paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= "+")))
> bosque <- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000  , cutoff=c(0.2,0.8),importance=TRUE)
> bosque

Call:
 randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12,      ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE) 
               Type of random forest: classification
                     Number of trees: 1000
No. of variables tried at each split: 12

        OOB estimate of  error rate: 15.81%
Confusion matrix:
     No Si class.error
No 3999  1     0.00025
Si  758 42     0.94750

As we see, the out of bag error for the level "SI" is 0.94750. However If I use the test set to get a sense of the error, the result is very different:

res.arbol <- predict(bosque,test.fullarbol,type="class")
> summary(res.arbol)
  No   Si 
7761   43 
> table(res.arbol,test.fullarbol$ex)

res.arbol   No   Si
       No 6937  824
       Si    4   39
> prop.table(table(res.arbol,test.fullarbol$ex),1)

res.arbol         No         Si
       No 0.89382811 0.10617189
       Si 0.09302326 0.90697674

Now we see the error rate for the test set in the class "Si" is very low , it's equal to 0.093 and It doesn´t make sense to me.

I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?

$\endgroup$
  • $\begingroup$ It could be test shows a very similar confusion matrix, just that it transposed. Neither the print.randomForest or table confusion matrix states clearly if rows are true class or predicted class. $\endgroup$ – Soren Havelund Welling Oct 4 '15 at 21:53
1
$\begingroup$

You have transposed the confusion matrix. See test example.

library(randomForest)
#simulate data set
set.seed(123)
obs = 4000+800+200
vars = 6
noise.factor=.2
X =  data.frame(replicate(vars,rnorm(obs)))
yvalue = X[,1]^2+sin(X[,2]) + rnorm(obs) * noise.factor
y = factor(c("No","Si")[(yvalue<=quantile(yvalue, 800/4800))*1+1])
test = sample(obs,200)
rf = randomForest(X[-test,],y[-test],ntree=50,cutoff=c(.2,.8))
print(rf)

#yep this confusion table is inverted compared to print.randomForest
table(predict(rf,X[test,]),y[test])

#here row's a true class, and columns are predicted class
table(y[test],predict(rf,X[test,],))

#cutoff is inherrited from training, but can also be modified during prediction
table(y[test],predict(rf,X[test,]))
table(y[test],predict(rf,X[test,],cutoff=c(.2,.8)))
table(y[test],predict(rf,X[test,],cutoff=c(.7,.000001)))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.