What causes the elbow shape of the loss curve? Looking at FractalNet and Resnet articles, I wonder what causes the loss curve to be shaped this way:


The learning speed plateaus and then suddenly drops significantly. What caught my eye was that this happens around round numbers of epochs, suggesting human factor.
And indeed for FractalNet I found this written in the article:

For CIFAR/SVHN, we drop the learning rate by a factor of 10 whenever
  the number of remaining epochs halves.

My question is, obviously especially in the case of FractalNet, the plateau has been there for quite some time before the 200 epoch mark. Wouldn't it be better to (a) reduce the learning rate sooner or (b) use some linear learning rate reduction scheme?
At least visually it seems that this would speed up the learning by 25% at the minimum.
Are there any reasons for not using this? Just the additional overhead of the added hyperparameters?
If it was only one paper I'd think they just didn't bother, but seeing this across the spectrum makes me suspicious that there is something greater that I'm missing.
 A: There are 3 reasons learning can slow, when considering the learning rate:  


*

*the optimal value has been reached (or at least a local minimum)

*The learning rate is too big and we are overshooting our target

*We are at a plateau with a very small gradient and the learning rate is too small to get us out there quickly.


Now one issue is determining in which situation you are. Would decreasing the learning rate help or make the problem worse?  And if we need to decrease the learning rate, how fast do we need to do it?
For FractalNet, we see that the performance for epochs 50-200 is quite poor, but for 1-50 is quite good. Undoubtedly decreasing the learning rate in a linear manner is going to decrease performance on those first 50 epochs. On the other hand, there might be a plateau that the algorithm found in epochs 50-200, so it is not certain that a lower learning rate would even improve the performance in those epochs. So what do we do? Do we reduce the learning rate linearly and if so how quickly? 
There are various ways to approach this problem. An interesting one is for example momentum based approaches, where we try to filter out repeated overshooting. The issue is that the perfect solution varies from problem to problem and it takes quite some testing to figure out which approach has the best effect. It then could be faster to just do something simple like this and train it a bit longer than to investigate the problem for weeks to reduce training time. 
