I'm a newbie learning Random Forest.

When I use this method to predict my outcome and check with the same data set (training set), I see that the model fits almost perfectly the data. But when I check with the test set, I obtain the good TPR (True Positive Ratio ~ 99%) but very bad FPR (False Positive Ratio ~ 64%).

Confusion table

What is the reason here? And what is the model-building strategies for Random Forest?

  • 1
    $\begingroup$ This is not unique to Random Forests, and I think you'll benefit in this and future projects from looking at why you have training and test sets and what "overfitting" means. $\endgroup$ – Wayne Jan 30 '17 at 22:57

Many default implementations of Random Forests (such as in scikit-learn) build decision trees that are very deep, which means that splits will continue to occur until only a single observation is in each terminal node (or until all the observations in a node have the same label).

To predict using a deep decision tree, an observation is sent down the tree and the average value of the training observations that created the terminal node is the value predicted by the tree.

When you use the model to predict an observation that was in your training data set, then the observation will land in the same terminal node it made when creating the tree. Because it is a deep tree, the average value of this terminal node is the same value of the training observation and your prediction will be perfect.

A Random Forest is simply a collection of deep decision trees. If bootstrap sampling were not used to create each tree, then a Random Forest would perfectly predict each training observation (because each tree would have used each observation and each individual tree would perfectly predict each observation).

Because of bootstrap sampling, about 2/3 of the observations are used to build each tree, which means that a majority of your trees will perfectly predict the correct labels. Because the final prediction is based on a majority vote, you can expect the final prediction to be correct each time as well.

A possible reason the predictions might not be exactly perfect is because due to the random nature of the bootstrap sample, some observations could be used in less than half of the trees and the other trees predict the incorrect class.

My advice: Don't even consider the training accuracy when using decision trees. Only consider out of sample (or out of bag) predictions.

The reason for the imbalance of TPR and FPR is based on what split point you used in your final prediction. Don't predict the class label. Instead predict the probability of the class belonging to a label (.predict_proba in scikit-learn). Then, you can choose a split point that leads to the TPR/FPR ratio you want.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.