8
$\begingroup$

Why is it that for Random Forest we take the average vote from each classifier in the ensemble rather than the average probability from each classifier in the ensemble? Is there theory behind why polling is preferred, or is it simply that it seems to work better empirically? Using the probabilities seems like it would eliminate the need for tuning the cutoff.

$\endgroup$
2
  • $\begingroup$ the trees in random forests are not pruned, so will you get a probability from each classifier? unless you're using one of the tricks to induce node impurity model = randomForest (y ~., data=data, nodesize=floor (0.1*nrow (data)) $\endgroup$
    – charles
    Sep 16, 2014 at 2:27
  • 2
    $\begingroup$ I'm thinking in the case where you aren't growing the trees to maximum depth so you have more than one observation in each node, why would we poll each tree for it's vote rather than simply averaging the probability prediction of each terminal node? Has Breiman or others published any theory behind the advantages of collapsing a weak learner's probability down to a single class vote before averaging across the ensemble? $\endgroup$
    – andrew
    Sep 16, 2014 at 13:57

1 Answer 1

2
$\begingroup$

This looks like the kind of answer you're looking for: http://people.dsv.su.se/~henke/papers/bostrom07c.pdf

The author looks at using average votes vs average probabilities from the ensemble members as well as a few other approaches to approximate impurity in the leaf nodes. For example, even if you do grow the trees to a maximum depth (as mentioned in the comments) the "Laplace Approximation" could be used to get a non-zero probability for each class by simply adding one to the count of observations of each class in the leaves.

Empirically speaking, the author concludes by saying that using averages of relative class frequencies (on 34 datasets) is better than using average votes (i.e. polling), though it is not better than using some "adjusted" probability average like the Laplace Approximation.

The difference looks pretty slight to me, but take a look at the "Accuracy and AUC" table on page 5. That might convince you one way or another.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.