Learning Rate impact on model building time

Question

I wanted to know that does learning rate impact the model building time in case of Gradient Boosted Trees. I do understand that increasing the number of trees have an impact( more the trees, more the time) however I am not sure about the learning rate.

For example, if I train my model with learning_rate = [0.1,0.2,0.3] how do these values rank up in terms of building time.

My understanding is that the lower the learning rate, the more the time gradient descent takes to converge and hence more building time. However, after reading Q23 from the below link, I am confused.

https://www.analyticsvidhya.com/blog/2017/09/30-questions-test-tree-based-models/

Can anyone please share some insight on this.

Answer 1

The learning rate $\eta$ is effectively the size of the step we are taking when we are moving along the gradient of our cost function. Note that $\eta$ depends on the number of iterations (steps) $M$ done and it helps us prevent overfitting. It works as a regularisation parameter too and that is why it is sometimes also referred to as shrinkage (e.g. in Hastie's et al. (2009) Elements of Statistical Learning in Sect. 10.12.1).

In general, "smaller" learning rates allow us to have smoother transitions between successive steps in our hypothesis set $H$ (the set of our candidate models). As Hastie et al. (2009) write: "empirically it has been found (Friedman, 2001) that smaller values of (the learning rate) $\nu$ favor better test error, and require correspondingly larger values of (number of iterations) M".

Regarding the question you are linking against: its context is a bit flawed (it does not make much sense to change $\eta$ without changing the number of iterations too); nevertheless if we are to do $M$ number of steps, the actual size of the step is not what costs us time. Calculating the direction of the step will cost us time but the actual cost of "taking the step" is largely similar irrespective of step size.