What are the benefits of layer-specific learning rates?

Question

I've read about using different learning rates for different layers of neural networks instead of using the same global learning rate for each layer.

What's the need for using these different learning rates specific to each layer?

Answer 1

Here is a great article on this topic: Basically, it usually makes training faster. The first layers are usually good enough, so the learning rate for them could be lower, but the last layers need to be tuned for out dataset, so lr should be higher.

Answer 2

The layer-specific learning rates help in overcoming the slow learning (thus slow training) problem in deep neural networks. As stated in the paper Layer-Specific Adaptive Learning Rates for Deep Networks:

When the gradient descent methods are used to train deep networks, additional problems are introduced. As the number of layers in a network increases, the gradients that are propagated back to the initial layers get very small (vanishing gradient problem). This dramatically slows down the rate of learning in the initial layers and slows down the convergence of the whole network.

The learning rates specific to each layer in the network allows larger learning rates to compensate for the small size of gradients in shallow layers (layers near the input layer).

---

This method also helps in transfer learning as stated in the answer by @Andrey based on the blog post. -- check discriminative fine-tuning by Jeremy Howard and post on CaffeNet.

The intuition is that in the layers closer to the input layer are more likely to have learned more general features -- such as lines and edges, which we won’t want to change much. Thus, we set their learning rate low. On the other hand, in case of later layers of the model -- which learn the detailed features, we increase the learning rate -- to let the new layers learn fast.