Why models often benefit from reducing the learning rate during training In Keras official documentation for ReduceLROnPlateau class they mention that

Models often benefit from reducing the learning rate

Why is that so? It's counter-intuitive for me at least, since from what I know- a higher learning rate allows taking further steps from my current position, and if I'll reduce the LR I might never "escape" a certain minimum.
 A: In ML methods where terms are added consequently (e.g. boosting) a learning rate below 1 is equivalent to shrinking coefficients of predictors towards 0. So reduced learning rate works for the same reason lasso and ridge regression work: regularization. In situations with high predictor-to-observation ratio, regularization reduces model variance at the expense of introducing a relatively small bias. Since the overall predictive error follows the rule
$$
\text{Mean-error Error} = \text{Bias}^2 + \text{Variance},
$$
the predictive error benefits from regularization.
A: A more complete quote is:

Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates.

(my emphasis).
There are two main reasons why your learning might seem to be stuck:

*

*You are around a local minimum and in every direction you might go you'd increase your error. In that case, as you correctly state, increasing the learning rate might allow you to "jump out" of the local minimum.


*You are in a steep valley and are oscillating between the walls. At one point the gradient points steep left; you make a large step in that direction and find yourself on the other side with the gradient pointing steep right. You are basically jumping back and forth far above the minimum. Here, decreasing the learning rate might allow you to descend into the minimum.
The above quote refers to this second case.
