2
$\begingroup$

I am new to ML, and just learned about the decision tree, and how entropy and information gain are used for a single tree. I am currently learning about random forest now, and some tutorials mention that a forest is formed from a number of trees, and at the end, majority voting is used on the collection of individual classification results by individual trees.

I am deeply grateful for your answers to the following:

  1. Apart from majority voting, does the Random Forest itself has entropy and information gain (or any other important measures)? I mean apart from entropy and information gain used for each tree in the forest, does the forest itself uses some type of entropy/information gain or any other measures apart from just counting votes?

  2. Could you refer a good resources for learning about details of Random forests and their applications?

$\endgroup$
1
  • 2
    $\begingroup$ Leo Breiman originated RFs in the late 90s as a response to criticisms that standalone CART trees (also his invention in the 80s) were predictively weak and unstable solutions. RFs integrated bootstrap resampling of both observations and features across many iterations or mini-CART trees, ensemble predictions were enabled. At the time of his first paper(s), entropy and information gain were not included as performance metrics, predictive accuracy was the criterion. stat.berkeley.edu/~breiman/randomforest2001.pdf $\endgroup$
    – user78229
    Jul 6, 2023 at 15:37

1 Answer 1

2
$\begingroup$

The term "Random Forest" is broad and describes simply an ensemble of randomized decision trees. "Ensemble" means that you somehow construct individual models (here: trees) and then somehow combine their predictions. "Randomized" means two things

  • Unlike Decision Trees, we do not consider all possibilities when looking for a split direction/dimension and threshold for a node. Instead, we pick at random a sample of candidate split directions and evaluate only these.
  • Each individual tree is grown not on the full dataset but a subsample of it.

The randomization means that individual trees will be diverse and uncorrelated. This is essentially the reason why Random Forests can outperform Decision Trees.

As you already indicated, a Random Forest prediction is the combination of the individual tree's predictions. This can be done using majority voting or another method, depending on the setup. Usually, it will be some form of mean, however (arithmetic, geometric, harmonic, ...).

How does majority voting fit in there? You can understand these means as centroids. And the majority vote is the centroid w.r.t. to the 0-1-Loss / symmetric difference metric.

So, no, as far as I know in conventional RFs there is no other mechanism that uses concepts of Entropy or Information Gain.

As for learning sources, one standard reference that is often recommended is "Elements of Statistical Learning", available online.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.