5
$\begingroup$

I'm using the R package randomForest. When you fit a model, it outputs the confusion matrix, but this completely mismatches what I find when I calculate the confusion matrix based on majority vote myself, using the model predictions. According to the documentation, the default is to use majority vote as the cutoff for classification, so I can't make sense of this.

Here is an example: 

require(randomForest)
set.seed(1)
y <- runif(500)<.5 
x <- matrix(rnorm(5000),500,10) 
z <- cbind(y,x)
colnames(z) <- c("y",paste("x",c(1:10),sep=""))

rfm <- randomForest( as.factor(y) ~ ., data=z ) 
rfm$confusion

    0   1 class.error
0  81 149   0.6478261
1 101 169   0.3740741

pred <- predict(rfm, z, type="vote", norm.votes=FALSE)[,2]
table(pred>250,y) # there are 500 trees, so >250 is a majority

   y
        FALSE TRUE
  FALSE   230    0
  TRUE      0  270

Any clue what is going on here?

|cite|improve this question
$\endgroup$
  • $\begingroup$ In the context of the answer, this thread seems statistical-enough to be considered on topic here, & to stay open, IMO. $\endgroup$ – gung - Reinstate Monica♦ Oct 2 '15 at 19:28
  • $\begingroup$ @gung: you do have a point there. I'm retracting my close vote. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 2 '15 at 19:37
  • 1
    $\begingroup$ To read the question, it sounds like a question about how to use R (ie off topic), @StephanKolassa--I would otherwise have voted w/ you. But your answer (+1 btw) suggests that a statistical confusion underlies the misunderstanding of the documentation. That's my take. $\endgroup$ – gung - Reinstate Monica♦ Oct 2 '15 at 19:41
4
$\begingroup$

Here is the relevant part of the help page ?randomForest:

confusion: (classification only) the confusion matrix of the prediction
          (based on OOB data).

Note the second parenthesis. The confusion output is derived from the out-of-bag data.

What does this mean? Part of what a random forest does is bootstrap the data, i.e., draw random samples with replacement from the original sample. In each instance, a model is fit to the data drawn. Then this model is applied to predict the data NOT drawn (the "out-of-bag" sample). This is a very smart trick to approximate the true expected out-of-sample error rate.

In contrast, what your second-to-last line does is that you apply the final model to all data, so you perform an in-sample fitting test. Of course this performs much better. However, in-sample accuracy is never a reliable guide to out-of-sample predictive accuracy. So I'd rather trust the confusion output of randomForest().

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you!!! So, for each tree that is fit as part of the "forest", some data is withheld? Is it clear how much data is withheld? Is this a "knob" the user can turn? $\endgroup$ – Rand Forrester Oct 5 '15 at 16:56
  • $\begingroup$ The amount of data "withheld" during bootstrapping is nondeterministic (since we sample at random) and can't be tuned. If the original data set has $n$ data points, then in bootstrapping, we sample with replacement $n$ points. So each original point has a chance of $(1-\frac{1}{n})^n$ of not being sampled in any particular bootstrap sample, which converges to $\frac{1}{e}$ as $n\to\infty$. Incidentally, the complement, $1-\frac{1}{e}\approx .632$, is the chance for any particular data point to be present in a bootstrap sample. This is the ".632 rule" you will often see in bootstrapping. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 6 '15 at 8:47
  • $\begingroup$ OK. I did not realize that re-sampling with replacement a data set of size $n$ was an inherent part of the algorithm. $\endgroup$ – Rand Forrester Oct 6 '15 at 16:15
  • $\begingroup$ It is. Along with only using a (small) random subset of your predictor variables. Which has two cool effects: (1) it reduces the effect of correlations between predictors, (2) it makes training a random forest much faster than training a simple tree with the full set of predictors. Random forests are a neat idea, indeed! $\endgroup$ – S. Kolassa - Reinstate Monica Oct 6 '15 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.