4
$\begingroup$

I am trying to understand the ways that the predictions of each Tree in a Random Forest or each Fern in Random Ferns are combined to form a single response.

I did read somewhere (reference missing) that each tree/ferns result is counted as a vote for the class with the highest posterior probability in that leaf. Then all votes are counted and the highest vote is return.

This sort of democratic approach does not sound statistically sound to me. During training not all leafs are filled with the same amount of samples/information. Therefore I would consider some leafs response more powerful than others.

How can I account for the power of each leaf? What is the theory behind it?

EDIT: Concrete example

Assume we have three weak classifiers. For a single sample/datapoint each classifier returns a posterior distribution (learned from training set):

classifier A says P(class1) = 6/10, P(class2) = 4/10, --> class1

Classifier B says P(class1) = 6/10, P(class2) = 4/10, --> class1

Classifier C says P(class1) = 1/100, P(class2) = 99/100 --> class2

So if we would just count the democratic vote, the total classification would yield: class1.

Yet both A and B are less secure about their vote opposed to classifier C. How do i account for this classifiers certainty? Further, how to i account for the number of samples which ended up in the specific leaf, yielding each posterior. Image that Classifier A and B both got their posterior distribution each from 10 training samples, Classifier C in the other hand got his posterior from 100 samples. (Here the number of samples are those which end up in the specific leaf of the decision tree. Yet all classifiers have been trained with the same amount of training data - assume 200 samples)

$\endgroup$
2
$\begingroup$

We had a bachelor thesis about majority votes of the trees vs. probability vote of the trees (in your example this would be (0.4 +0.4 +0.99) / 3 = 0.5966667 --> class 2) at our faculty and the latter provided slightly better results in some simulations, but the difference between these two kind of votes is in general not very big, so it is not a problem to stick with the majority vote.

For the second question I would say, that you should not prefer trees with more or less samples in the leaf nodes, as this does not say anything about the certainity of your prediction. But this is also an interesting question.

$\endgroup$
  • $\begingroup$ Danke. Do you have any more Input / reading recommendation regarding this specific topic? $\endgroup$ – Nikolas Rieble Dec 1 '16 at 13:09
  • $\begingroup$ No, sorry I do not know any other resources. To give some numbers: In most of our examples the difference of the AUC's were around 0.001, so there seems not a lot to gain from the alternative approach. In very unbalanced cases (90 % of one class, 10 % of the other), the difference was higher and a gain of 0.01 of the AUC could be achieved. $\endgroup$ – PhilippPro Dec 1 '16 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.