4
$\begingroup$

Say we have a data frame df where diagnosis is the first column. There are only 2 possible values for diagnosis: C and N.

We fit a random forest:

rf <- randomForest(diagnosis ~ ., data=df)

We then extract probabilities in two different ways:

probs1 = rf$votes
probs2 = predict(rf, df[,-1], type="prob" )

So if we now examine these extracted probabilities:

> head(probs1)
                     C          N
sample1220  0.52095808 0.47904192
sample1831  0.06698565 0.93301435
sample526   0.15286624 0.84713376
sample1225  0.30808081 0.69191919
sample523   0.03910615 0.96089385
sample644   0.97093023 0.02906977

> head(probs2)
                C     N
sample1220  0.174 0.826
sample1831  0.028 0.972
sample526   0.734 0.266
sample1225  0.726 0.274
sample523   0.014 0.986
sample644   0.990 0.010

Why are the probabilities in probs2 (from the predict function) different from the ones in probs1 (which are the proportions of votes) ?

I thought the probabilities were the proportions of votes, but clearly that's not the case. Could anybody please explain?

$\endgroup$
2
  • $\begingroup$ What discrepancy have you observed? Can you share examples of the discrepant predictions? There's not much to say with so little information. I suspect you're describing the same issue as here: stats.stackexchange.com/questions/405234/… If you can't share your data, you can use a publicly-available data set to share your findings. $\endgroup$
    – Sycorax
    Jun 11, 2019 at 23:35
  • $\begingroup$ I was confused by the terminology. You're talking about "merging" accounts, but I've only ever registered one account. When I originally posted my question, I clicked on the "post as guest" option, so there's an inconsistency in the terminology there. Thanks very much for your kind help. $\endgroup$
    – Dave
    Jun 13, 2019 at 18:02

1 Answer 1

2
$\begingroup$

The observed behavior in the post is expected behavior, and essentially a duplicate of Random Forest different results for same observation

The discrepancy you find is fully explained by understanding what the software is doing. Attribute rf$votes reports OOB results for the training data, while calling predict(rf, newdata=something, ...) (which is syntactically equivalent to predict(rf,something, ...)) reports predictions for the whole ensemble, without regard to in- or out-of-bag status. The two results would agree if you did not supply argument newdata in the call to predict, as in predict(rf,type="prob").

From the documentation:

votes (classification only) a matrix with one row for each input data point and one column for each class, giving the fraction or number of (OOB) `votes' from the random forest.

Where as for the predict method we have

newdata a data frame or matrix containing new data. (Note: If not given, the out-of-bag prediction in object is returned.)

So putting these two facts together, we can conclude that supplying the argument newdata is giving the random forest's predictions including all of the trees, while votes is only reporting OOB results. Obviously, these wouldn't agree in general, so that's why your results disagree.

$\endgroup$
9
  • $\begingroup$ Thank you very much Sycorax for your answer. Much appreciated. Can I please check if I understand what you're saying? Are you saying that the probabilities are different because the probabilities that come from predict are computed using the votes from all the trees in the forest, whereas the probabilities in $votes are computed using only the votes of the trees who never saw the corresponding rows of the dataframe during training? Thanks! $\endgroup$
    – Dave
    Jun 13, 2019 at 17:59
  • $\begingroup$ The operation of predict changes depending on whether or not you supply newdata as an argument to the call. If you don't supply a newdata argument, you get OOB predictions from the training data. If you do supply newdata, then you get predictions from the whole ensemble. rf$votes reports OOB results for the training data. In any case, the software doesn't make any sort of comparison to determine if any provided data matches the original training data. $\endgroup$
    – Sycorax
    Jun 13, 2019 at 18:04
  • $\begingroup$ I saw that's what you explained in the other thread, and I don't fully understand it because I don't fully understand what "out of bag" means. I've been looking for a definition, but every time I look it up, I find stuff about "out of bag error", but I haven't been able to find a definition of what OOB itself means. Could you please give me a definition of OOB or point me to somewhere where I can read what exactly OOB means? Thanks very much. $\endgroup$
    – Dave
    Jun 13, 2019 at 18:10
  • $\begingroup$ Each tree in a random forest is grown using a bootstrap sample of the data. The data in the bootstrap is in bag. The data that is not in the bootstrap is out of bag. $\endgroup$
    – Sycorax
    Jun 13, 2019 at 18:44
  • $\begingroup$ so the probability in $votes for each row is computed only with the votes of the trees for which that row is out of bag? $\endgroup$
    – Dave
    Jun 14, 2019 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.