# Why does randomForest confusion matrix not match the one I calculate using predictions from the model object?

I'm using the R package randomForest. When you fit a model, it outputs the confusion matrix, but this completely mismatches what I find when I calculate the confusion matrix based on majority vote myself, using the model predictions. According to the documentation, the default is to use majority vote as the cutoff for classification, so I can't make sense of this.

Here is an example:

require(randomForest)
set.seed(1)
y <- runif(500)<.5
x <- matrix(rnorm(5000),500,10)
z <- cbind(y,x)
colnames(z) <- c("y",paste("x",c(1:10),sep=""))

rfm <- randomForest( as.factor(y) ~ ., data=z )
rfm$confusion 0 1 class.error 0 81 149 0.6478261 1 101 169 0.3740741 pred <- predict(rfm, z, type="vote", norm.votes=FALSE)[,2] table(pred>250,y) # there are 500 trees, so >250 is a majority y FALSE TRUE FALSE 230 0 TRUE 0 270  Any clue what is going on here? • In the context of the answer, this thread seems statistical-enough to be considered on topic here, & to stay open, IMO. – gung - Reinstate Monica Oct 2 '15 at 19:28 • @gung: you do have a point there. I'm retracting my close vote. – S. Kolassa - Reinstate Monica Oct 2 '15 at 19:37 • To read the question, it sounds like a question about how to use R (ie off topic), @StephanKolassa--I would otherwise have voted w/ you. But your answer (+1 btw) suggests that a statistical confusion underlies the misunderstanding of the documentation. That's my take. – gung - Reinstate Monica Oct 2 '15 at 19:41 ## 1 Answer Here is the relevant part of the help page ?randomForest: confusion: (classification only) the confusion matrix of the prediction (based on OOB data).  Note the second parenthesis. The confusion output is derived from the out-of-bag data. What does this mean? Part of what a random forest does is bootstrap the data, i.e., draw random samples with replacement from the original sample. In each instance, a model is fit to the data drawn. Then this model is applied to predict the data NOT drawn (the "out-of-bag" sample). This is a very smart trick to approximate the true expected out-of-sample error rate. In contrast, what your second-to-last line does is that you apply the final model to all data, so you perform an in-sample fitting test. Of course this performs much better. However, in-sample accuracy is never a reliable guide to out-of-sample predictive accuracy. So I'd rather trust the confusion output of randomForest(). • Thank you!!! So, for each tree that is fit as part of the "forest", some data is withheld? Is it clear how much data is withheld? Is this a "knob" the user can turn? – Rand Forrester Oct 5 '15 at 16:56 • The amount of data "withheld" during bootstrapping is nondeterministic (since we sample at random) and can't be tuned. If the original data set has$n$data points, then in bootstrapping, we sample with replacement$n$points. So each original point has a chance of$(1-\frac{1}{n})^n$of not being sampled in any particular bootstrap sample, which converges to$\frac{1}{e}$as$n\to\infty$. Incidentally, the complement,$1-\frac{1}{e}\approx .632$, is the chance for any particular data point to be present in a bootstrap sample. This is the ".632 rule" you will often see in bootstrapping. – S. Kolassa - Reinstate Monica Oct 6 '15 at 8:47 • OK. I did not realize that re-sampling with replacement a data set of size$n\$ was an inherent part of the algorithm. – Rand Forrester Oct 6 '15 at 16:15
• It is. Along with only using a (small) random subset of your predictor variables. Which has two cool effects: (1) it reduces the effect of correlations between predictors, (2) it makes training a random forest much faster than training a simple tree with the full set of predictors. Random forests are a neat idea, indeed! – S. Kolassa - Reinstate Monica Oct 6 '15 at 16:21