I have recently done some work altering popular gradient boosted decision trees (GBDTs) for regression, and I was just working on establishing a theoretic basis for the modern algorithm. There is a paper here I've gravitated toward from Jerome Friedman (1999): Greedy Function Approximation: A Gradient Boosting Machine. There is a nice background developed in the intro, and I've found what I think is the general algorithm for GBDTs for regression, copied here, Algorithm 2.
My question is going to become apparent now. Inside the for
loop we have three lines. The first line is simply computing the gradients based on the stagewise learner until that step. The $h(\textbf{x},\textbf{a})$ is the parametrized function (regression tree) that is fit to the residuals at each stage. This fitting is happening on the second line. On this line the parameter $\rho_m$ is also being optimized, which is then used in the last step to scale the update to the function $F$.
The way I interpret $\rho_m$ is what we now call the learning rate in popular GBDT libraries like XGBoost or LightGBM. $\rho_m$ is set as a fixed value at the beginning of training, it is not optimized for at each step like in this algorithm. Why is that? Is there an important paper that discusses why this simplification is made? This algorithm makes it seem that we should adjust the learning rate at each stage instead of keeping it fixed. I'm just wondering why we seem to have lost this detail with modern GBDT implementations?