I was reading about learning rate change in a machine learning course. The following statement and image appear:

Turning down the learning rate reduces the random fluctuations in the error due to the different gradients on different mini-batches.

enter image description here

So why if we reduce the learning rate, the error is quickly reduced afterwards?

Also, why does the smaller learning rate converge slower after that?


1 Answer 1


I think an intuitive answer goes along these lines:

Suppose you want to find a parameter that 'best explains' your data in only one dimension. 'Best explains' means to make an error function small (or a likelihood big, these are usually the same as error = -log(likelihood)). We forget for a moment that we want to talk about neural nets: our model is very simplistic: its just $f(x,\theta) = x + \theta$. $x$ is the input for the function and $\theta$ is the parameter of the model (in the case of NNs its the weights). We want to find the best parameter for explaining the single data point $(x,y) = (1,5)$. Also imagine that we do not use an L2-error but an L1-error, i.e. the error function is

$$l(\hat{y}, y) = |\hat{y} - y| = |x + \theta - y|$$

Of course, the absolute value is not differentiable at $0$ so we just define the differential of the absolute value at $0$ to be $0$ (if you are exactly at the minimal value of the function, then do not move at all). Otherwise the differential is $+1$ if $x + \theta < y$ and $-1$ if $x+\theta > y$. We start with some random $\theta$, for example, $\theta = -1.5$. Then $\hat{y} = x + \theta = 1 - 1.5 = -0.5$ so graphically, the error function looks like this:

enter image description here

Now the derivative is $+1$ so we move $\theta$ by $+1$ to the right which makes $\hat{y} = +0.5$:

enter image description here

After that the derivative is $-1$ which makes us set $\theta$ back to $-0.5$ and so on and so on. We always run around the minimal value but since we do not give any weight to our gradient, we circulate around it instead of finding our way inside. This is intuitively what could happens when you set your learning rate to be constantly $1$ instead of some lower value. Of course, in the real world we use an L2-error which already would take the distance to the minimal value into account and everything looks much more complicated... Also the reason is not that we are principally unable to find our way to the minimal value but rather that there are numerical instabilities which makes us over- or underestimate the 'perfect' gradient.

Question 2): The only explanation I can come up with is the following: Suppose we use a constant learning rate and the parameter search algorithm spirals around a local minimum for some time:

enter image description here

Sue to the fact that the learning rate stays constant it could happen that at some point the algorithm 'spiraled' into a suitable point so that in the next step it actually overshoots so much that it moves out of the complete region it was in all the time and 'accidentally' finds its way into a completely new valley and into the direction of another even lower local minimum that it can spiral around:

enter image description here

However, despite all of this I would not pay too much attention to question no. 2 in general: What the error graph does not tell you is that after so many epochs you are probably overfitting your data quite a lot, i.e. the best solution for your problem is mostly not the global minimum of the error function! Question no. 1 indeed is much more important. You will find realizations of the concept of an 'adaptive' learning rate in many different algorithms (for example in Gradient Boosting which is completely unrelated to NNs you have trees and you do not simply add the next tree but you add $\lambda*\text{next tree}$ where $0 < \lambda < 1$).

  • $\begingroup$ Then why does the smaller learning rate converge slower? $\endgroup$
    – bsky
    May 30, 2017 at 16:32
  • $\begingroup$ Just to understand your question: you do not ask about convergence, right? Because convergence in this practical sense means "how fast the algorithm reaches its self defined goal"... you ask why the error is higher with a smaller learning rate after many epochs (I.e. you ask why the green curve finally goes below the red curve)... did I understand correctly? $\endgroup$ May 31, 2017 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.