I have a question about Random Forest algorithm:
1.Let the number of training cases be N, and the number of variables in the classifier be M.
2.We are told the number m of input variables to be used to determine the decision at a node of the tree; m should be much less than M.
3.Choose a training set for this tree by choosing n times with replacement from all N available training cases (i.e., take a bootstrap sample). Use the rest of the cases to estimate the error of the tree, by predicting their classes.
4.For each node of the tree, randomly choose m variables on which to base the decision at that node. Calculate the best split based on these m variables in the training set.
5.Each tree is fully grown and not pruned (as may be done in constructing a normal tree classifier).
I think I understand the most steps, except step 3. How do we in practice predict the classes? Does anyone have a simple explanation? Because I cant find any explanation online... I understand that :
Finally a response class is predicted in each terminal node of the tree (or each rectangular section
in the partition respectively) by means of deriving from all observations in this node either the
average response value in regression or the most frequent response class in classification trees. Note
that this means that a regression tree creates a piecewise (or rectangle-wise for two dimensions
and cuboid-wise in higher dimensions) constant prediction function.
But I am trying to understand How do we use Out Of Bag data later for finding prediction error?
For example I am trying to understand In an ensemble of trees the predictions of all individual trees need to be combined. This is
usually accomplished by means of (weighted or unweighted) averaging in regression or voting in
classification.
The term “voting” can be taken literally here: Each subject with given values of the predictor
variables is “dropped through” every tree in the ensemble, so that each single tree returns a
predicted class for the subject. The class that most trees “vote” for is returned as the prediction
of the ensemble. This democratic voting process is the reason why ensemble methods are also
called “committee” methods. Note, however, that there is no diagnostic for the unanimity of the
vote. For regression and for predicting probabilities, i.e. relative class frequencies, the results of
the single trees are averaged; some algorithms also employ weighted averages.
Best Regards!
