# How are individual trees added together in boosted regression tree?

I'm reading Introduction to Statistical Learning, James, G., et al. (2013), in which they describe the Boosted Regression Tree algorithm as following. What I do not understand is Eq 8.10 and 8.11. What do "adding the new tree to the old tree" & "update the residual", as signified by $\leftarrow$, mean mathematically?

The working guide to boosted regression tree by the same authors do not explain how exactly to add the trees either.

• You don't actually "sum trees" -- the pseudocode you have there clearly indicates what you do sum. Are you clear on what $f$ and $f^b$ are? Jan 29, 2015 at 1:09
• You seem to be referring to James, G., et al. (2013) not Hastie, T., et al. (2011). I frequently confuse them myself. Jan 29, 2015 at 1:09
• @Glen_b Thanks for pointing out a detail I miss. So $f$ is the tree, but $\hat f(x)$ is the predicted value after plugging $x$ into the tree's decision rule. So (8.11) makes sense because $r_i$ and $\hat f^b(x_i)$ are both a number. But (8.10) is unclear: how am I updating the tree (a set of decision rules, not any particular value), while the pseudocode is saying $\text{a number} \leftarrow \text{a number} + \lambda \text{a number}$? Jan 29, 2015 at 3:01
• @user12202013 Both James and Hastie are authors on the book, so while it would be unconventional to refer to the book as Hastie et al rather than James et al, I wouldn't say it's actually wrong since Hastie is an author. Jan 29, 2015 at 3:03
• Strictly speaking I wouldn't call "$f$" exactly a tree either. $f$ is the underlying function we're trying to fit by using a tree-model. When we speak of something being "a tree" we're usually using shorthand. Jan 29, 2015 at 3:05

They assume that you're keeping track of a "current estimator" $\hat f$, which is a sum of all the trees you've seen so far. (In code you would just store this as an array of all the trees you've seen so far.) The $\leftarrow$ sign just means "takes the new value"--so when they say "add the new tree" they mean, basically, append the new tree to the array of trees you already store, so that where you previously would have computed $\hat f$ with that array, you now compute $\hat f + \lambda \hat f^b$. (The $\leftarrow$ sign just means "takes the new value".)
The residual is just the difference between the response and your current prediction $\hat f$. So if you add something to $\hat f$ you need to subtract it from the residual so that they continue to sum up to the target response. Again, the $\leftarrow$ sign just means "takes the new value"--so $r_i \leftarrow r_i - \lambda \hat f^b(x_i)$ would translate in code to r[i] -= lambda * tree_prediction[i] or something.
• As the algorithm states, the final model is $\hat f = \sum \lambda \hat f^b$--in other words, you just sum the values that each individual tree gives (times the learning rate). Jan 29, 2015 at 0:49