I was reading about learning rate change in a machine learning course. The following statement and image appear:
Turning down the learning rate reduces the random fluctuations in the error due to the different gradients on different mini-batches.
So why if we reduce the learning rate, the error is quickly reduced afterwards?
Also, why does the smaller learning rate converge slower after that?
I think an intuitive answer goes along these lines:
Suppose you want to find a parameter that 'best explains' your data in only one dimension. 'Best explains' means to make an error function small (or a likelihood big, these are usually the same as error = -log(likelihood)). We forget for a moment that we want to talk about neural nets: our model is very simplistic: its just $f(x,\theta) = x + \theta$. $x$ is the input for the function and $\theta$ is the parameter of the model (in the case of NNs its the weights). We want to find the best parameter for explaining the single data point $(x,y) = (1,5)$. Also imagine that we do not use an L2-error but an L1-error, i.e. the error function is
$$l(\hat{y}, y) = |\hat{y} - y| = |x + \theta - y|$$
Of course, the absolute value is not differentiable at $0$ so we just define the differential of the absolute value at $0$ to be $0$ (if you are exactly at the minimal value of the function, then do not move at all). Otherwise the differential is $+1$ if $x + \theta < y$ and $-1$ if $x+\theta > y$. We start with some random $\theta$, for example, $\theta = -1.5$. Then $\hat{y} = x + \theta = 1 - 1.5 = -0.5$ so graphically, the error function looks like this:
Now the derivative is $+1$ so we move $\theta$ by $+1$ to the right which makes $\hat{y} = +0.5$:
After that the derivative is $-1$ which makes us set $\theta$ back to $-0.5$ and so on and so on. We always run around the minimal value but since we do not give any weight to our gradient, we circulate around it instead of finding our way inside. This is intuitively what could happens when you set your learning rate to be constantly $1$ instead of some lower value. Of course, in the real world we use an L2-error which already would take the distance to the minimal value into account and everything looks much more complicated... Also the reason is not that we are principally unable to find our way to the minimal value but rather that there are numerical instabilities which makes us over- or underestimate the 'perfect' gradient.
Question 2): The only explanation I can come up with is the following: Suppose we use a constant learning rate and the parameter search algorithm spirals around a local minimum for some time:
Sue to the fact that the learning rate stays constant it could happen that at some point the algorithm 'spiraled' into a suitable point so that in the next step it actually overshoots so much that it moves out of the complete region it was in all the time and 'accidentally' finds its way into a completely new valley and into the direction of another even lower local minimum that it can spiral around:
However, despite all of this I would not pay too much attention to question no. 2 in general: What the error graph does not tell you is that after so many epochs you are probably overfitting your data quite a lot, i.e. the best solution for your problem is mostly not the global minimum of the error function! Question no. 1 indeed is much more important. You will find realizations of the concept of an 'adaptive' learning rate in many different algorithms (for example in Gradient Boosting which is completely unrelated to NNs you have trees and you do not simply add the next tree but you add $\lambda*\text{next tree}$ where $0 < \lambda < 1$).
