I'm reading ISLR and I don't understand what is the problem that random forests solve; what problems does tree correlation cause when using bagging?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ This is a useful thread: stats.stackexchange.com/questions/18891/… $\endgroup$– mktAug 2, 2017 at 19:06
-
$\begingroup$ The more trees, the less likely they all make the same error. With total correlation, you have one tree. $\endgroup$– Sam WeisenthalAug 2, 2017 at 19:59
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
I would like to answer this question by first overviewing bagging.
In bagged trees, we resample observations from a dataset with replacement and fit a tree. We consider all the features in our resampling and this process is repeated $n$ times. If you have ever fit a simple decision tree holding out a test set you will see that your results vary dramatically, every time you perform a training, testing split. This high variance is undesirable and therefore we consider a new dataset which is a subset of the original (bootstrap sample). We aggregate all the $n$ trees by averaging in a regressor or by majority vote in a classifier to obtain a final result.
One issue we have not considered in this bagging process is how similar the trees tend to be. While there are mathematical definitions to the correlation between these trees, consider this example.
Consider one strong predictor in our data set which reduces a measure of error (ex: RSS) the most. All our bagged trees tend to to make the same cuts because they all share the same features. This makes all these trees look very similar hence increasing correlation.
To solve tree correlation we allow random forest to randomly choose only $m$ predictors in performing the split. Now the bagged trees all have different randomly selected features to perform cuts on. Therefore, the feature space is split on different predictors, decorrelating all the trees.
When performing random forest if you set max_features=# features in the dataset (in scikit learn, mtry in R) you will be constructing a bagged decision tree model. If max_features<# of features we will be performing Random forest. It is always a good idea to tune this parameter in constructing any model.
-
1$\begingroup$ Thanks for the overview, but I'm still wondering why tree correlation is an issue at all? I understand that trees will be similar if there's one strong predictor in the dataset, but if it's such a good predictor, why does it pose a problem that all trees use it? $\endgroup$ Aug 3, 2017 at 13:11
-
2$\begingroup$ If all the trees as similar there is no point in performing random forest because all the bagged trees will look very similar, ie. similar to performing a single tree.Therefore, We have not addressed the original issue of variance. By decorrelating the trees and bagging we most effectively reduce the variance in our model. $\endgroup$– Sada93Aug 3, 2017 at 14:17
-
3$\begingroup$ Intuitively, I imagine a situation with two important predictors, one (Predictor1) having a minor but non-negligible advantage on the other (Predictor2) for reduction in (say) RSS. Then even if you bootstrap your sample, most trees are going to choose Predictor1 for the first split. It might be the case, however, that splitting first on Predictor2 leads to a different tree structure which captures some feature of the data not explained by a "Predictor1 Tree". So randomizing the available predictors at each split is a way to give the other predictors more of a chance. $\endgroup$– klumbardAug 3, 2017 at 15:31
-
$\begingroup$ @klumbard Okaaay, I understand better now! If I understand correctly, we actually want this randomization of predictor selection in order to see if different paths could lead to better trees. If you write your comment as an answer, I'll select it :) $\endgroup$ Aug 3, 2017 at 18:28
-
$\begingroup$ "Better" trees might not be the most apt term, but trees that are somehow encoding a different feature of the data not represented by the other tree. At the end of the random forest algorithm, we send the new sample down each of the trees we have created and then average the results of all trees for our final prediction. So have various uncorrelated trees allows us to pull out (different) relevant information from each of them and then re-combine at the end. My first comment is really not much different than the 4th and 5th paragraph of Sada93's answer, so IMO you should just accept that one. $\endgroup$– klumbardAug 3, 2017 at 18:34