# How are individual trees added together in boosted regression tree?

I'm reading Introduction to Statistical Learning, James, G., et al. (2013), in which they describe the Boosted Regression Tree algorithm as following. What I do not understand is Eq 8.10 and 8.11. What do "adding the new tree to the old tree" & "update the residual", as signified by $\leftarrow$, mean mathematically? The working guide to boosted regression tree by the same authors do not explain how exactly to add the trees either.

• You don't actually "sum trees" -- the pseudocode you have there clearly indicates what you do sum. Are you clear on what $f$ and $f^b$ are? – Glen_b Jan 29 '15 at 1:09
• You seem to be referring to James, G., et al. (2013) not Hastie, T., et al. (2011). I frequently confuse them myself. – Ellis Valentiner Jan 29 '15 at 1:09
• @Glen_b Thanks for pointing out a detail I miss. So $f$ is the tree, but $\hat f(x)$ is the predicted value after plugging $x$ into the tree's decision rule. So (8.11) makes sense because $r_i$ and $\hat f^b(x_i)$ are both a number. But (8.10) is unclear: how am I updating the tree (a set of decision rules, not any particular value), while the pseudocode is saying $\text{a number} \leftarrow \text{a number} + \lambda \text{a number}$? – Heisenberg Jan 29 '15 at 3:01
• @user12202013 Both James and Hastie are authors on the book, so while it would be unconventional to refer to the book as Hastie et al rather than James et al, I wouldn't say it's actually wrong since Hastie is an author. – Glen_b Jan 29 '15 at 3:03
• Strictly speaking I wouldn't call "$f$" exactly a tree either. $f$ is the underlying function we're trying to fit by using a tree-model. When we speak of something being "a tree" we're usually using shorthand. – Glen_b Jan 29 '15 at 3:05

They assume that you're keeping track of a "current estimator" $\hat f$, which is a sum of all the trees you've seen so far. (In code you would just store this as an array of all the trees you've seen so far.) The $\leftarrow$ sign just means "takes the new value"--so when they say "add the new tree" they mean, basically, append the new tree to the array of trees you already store, so that where you previously would have computed $\hat f$ with that array, you now compute $\hat f + \lambda \hat f^b$. (The $\leftarrow$ sign just means "takes the new value".)
The residual is just the difference between the response and your current prediction $\hat f$. So if you add something to $\hat f$ you need to subtract it from the residual so that they continue to sum up to the target response. Again, the $\leftarrow$ sign just means "takes the new value"--so $r_i \leftarrow r_i - \lambda \hat f^b(x_i)$ would translate in code to r[i] -= lambda * tree_prediction[i] or something.
• As the algorithm states, the final model is $\hat f = \sum \lambda \hat f^b$--in other words, you just sum the values that each individual tree gives (times the learning rate). – Ben Kuhn Jan 29 '15 at 0:49