7
$\begingroup$

I need some help understanding the concept of random forests. As I understand, when I make a decision tree, I carefully select each node so as to maximize the information gain and minimize the entropy, i.e each node should result in a higher information gain than its parent node.

If that is true, then the decision tree is already the best possible learner. Why do I need to combine it with other trees that may not be as good and then take a vote?

If I created the tree to maximize the information gain, then this is already the 'best' model.

I would understand the need for a random forest if I created 10 decision trees by randomly selecting the nodes to split on.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

The short answer is variance. The long answer is variance and also generalization. Decision trees have high variance: a slight change in the training data can cause a big change in how the splits occur, and therefore the predictions aren't very stable. Before RF, there was a lot of attention paid to pruning decision trees and so on to get better generalization. But by taking the average of many i.i.d. decision trees (a random forest), that variation is averaged out, and we get to have our cake and eat it by having a low bais, low variance classifier with excellent out-of-sample generalization.

This is explained in more detail in Elements of Statistical Learning.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks. So how do we construct these multiple trees. As I understand, only one can have the lowest entropy. Do we consider entropy and information gain at all or randomly create the the trees? $\endgroup$
    – Victor
    Sep 24, 2015 at 18:43
  • $\begingroup$ I am trying to understand how are the individual decision trees built? a.k.a How are the nodes selected? At random? $\endgroup$
    – Victor
    Sep 24, 2015 at 18:54
  • $\begingroup$ Ah, thanks. I am trying to understand how to build the decision trees that form a random forest $\endgroup$
    – Victor
    Sep 24, 2015 at 18:57
  • 1
    $\begingroup$ The gist of it is that you bootstrap your data for each tree and then build a decision tree only using $m$ randomly-selected features at a time, where $m$ is a training parameter. Then select a new bootstrap sample for your next tree and so on. (This is discussed at length in ESL.) $\endgroup$
    – Sycorax ♦
    Sep 24, 2015 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.