I'm reading ISLR and I don't understand what is the problem that random forests solve; what problems does tree correlation cause when using bagging?
I would like to answer this question by first overviewing bagging.
In bagged trees, we resample observations from a dataset with replacement and fit a tree. We consider all the features in our resampling and this process is repeated $n$ times. If you have ever fit a simple decision tree holding out a test set you will see that your results vary dramatically, every time you perform a training, testing split. This high variance is undesirable and therefore we consider a new dataset which is a subset of the original (bootstrap sample). We aggregate all the $n$ trees by averaging in a regressor or by majority vote in a classifier to obtain a final result.
One issue we have not considered in this bagging process is how similar the trees tend to be. While there are mathematical definitions to the correlation between these trees, consider this example.
Consider one strong predictor in our data set which reduces a measure of error (ex: RSS) the most. All our bagged trees tend to to make the same cuts because they all share the same features. This makes all these trees look very similar hence increasing correlation.
To solve tree correlation we allow random forest to randomly choose only $m$ predictors in performing the split. Now the bagged trees all have different randomly selected features to perform cuts on. Therefore, the feature space is split on different predictors, decorrelating all the trees.
When performing random forest if you set max_features=# features in the dataset (in scikit learn, mtry in R) you will be constructing a bagged decision tree model. If max_features<# of features we will be performing Random forest. It is always a good idea to tune this parameter in constructing any model.