My understanding of Random Forest: So Random Forest is an ensemble method. It consists of many trees. Essentially, we want each individual tree to be independent or uncorrelated from one another. We do so by injecting randomness into each tree. We pick random features to build out each tree. At the end, each tree will be different (because each tree will only have been exposed to random subset taken from the whole set of features), therefore there will be variations amongst each tree models. Ultimately we average all the trees together and we get a final model that closely resemble the actual model (label).
My understanding of Gradient Boosting: On the other hand, in gradient boosting, instead of creating multiple trees that independently model the data, it creates decision trees that build on each other. The goal of each new tree is to fix the errors where the previous trees are the most wrong. Decision trees are therefore grown sequentially, as each tree is created using information derived from the previous decision tree.
Quote I read from a Quora answer: “The way this works is that mistakes incurred on the training data are recorded and then applied to the next round of training data. At each iteration, weights are added to the training data based on the results of the previous iteration. Higher weighting is applied to instances that were incorrectly predicted from the training data and instances that were correctly predicted receive less weight. The training and test data are then compared and errors are again logged in order to inform weighting at each subsequent round. Earlier iterations that do not perform well, and that perhaps misclassified data, can thus be improved upon via further iterations.”
I understand the idea behind random forest, but it’s a little hard for me to make gradient boosting concrete in my head. I guess I’m mostly confused as to how...
“The goal of each new tree is to fix the errors where the previous trees are the most wrong.”
“At each iteration, weights are added to the training data based on the results of the previous iteration. Higher weighting is applied to instances that were incorrectly predicted from the training data and instances that were correctly predicted receive less weight.”
Can anyone create a very small and simple decision tree using gradient boosting and explain how the next tree minimizes the errors from the previous tree?