Relation between learning rate and number of hidden layers?

Question

Is there any rule of thumb between depth of a neural network and learning rate? I have been noticing that the deeper the network is, the lower the learning rate must be.

If that's correct, why is that?

Answer 1

This question has been answered here:

With neural networks, should the learning rate be in some way proportional to hidden layer sizes? Should they affect each other?

Short answer is yes, there is a relation. Though, the relation is not this trivial, all I can tell you that what you see is because the optimization surface becomes more complex as the the number of hidden layers increase, therefore smaller learning rates are generally better. While stucking in local minima is a possibility with low learning rate, it's much better than complex surface and high learning rate.

Answer 2

This is true, for instance, in linear networks. Consider the following product of $w\times w$ matrices

$$y=x W_1\ldots W_d$$

Now suppose our target labels are all 0's and we fit using least-squares loss.

If we initialize $W_i$ with IID $\mathcal N(0, 1/w)$, which is scale required to keep intermediate activations from exploding or shrinking to 0, trace of our loss Hessian w.r.t to $x$ is $O(d)$ by using this result.

Hence we must divide learning rate by $d$, the number of layers, to remain convergent.

Here's a plot of Hessian eigenvalues as function of depth for a randomly initialized linear neural network:

notebook

Answer 3

It is correct that deeper networks "prefer" lower learning rates. In general at least.

The common training method of backpropagation-with-gradient-descent, for example, works by calculating the impact of each weight on the final outcome - i.e. if you wiggle the weight, how fast does the error change - and using this to home in on the weights that, if changed, will give most bang for the buck. Mathematically:

$$ w \leftarrow w - \lambda \times \frac{dE}{dw} $$

This change is typically applied to all weights in all levels of the network, in parallel. So if you have three active layers - two hidden, one output - then, after a single datum's worth of gradient descent, your input data is now being multiplied by a set of corrected weights... then another set of corrected weights... then a third set of corrected weights. The correction is effectively triple-counted, give or take an activation function.

This is typically not a huge concern for practitioners because, unlike e.g. the Newton-Raphson method, gradient descent is far more focused on choosing which weights to change rather than ensuring that those weights are changed by a sensible amount. Compare the above formula with the one for Newton-Raphson optimisation:

$$ w \leftarrow w - \frac{E}{\frac{dE}{dw}} $$

Notice how the $ \frac{dE}{dw} $ appears in the denominator of this one rather than the numerator? Yeah, gradient descent is kinda the wrong way up... It's a system where doubling all your inputs and actual outputs can effectively quadruple your learning rate, despite these things being conceptually unrelated.

So, under anything resembling gradient descent, if your learning rate is high enough that increasing the layer count can cause you significant trouble, then frankly it's far too high and you've probably already got bigger problems!

---

Notes:

(1) This is a different issue from the one in the post that NULL linked to, which was about the number of neurons in a (single) hidden layer rather than the number of layers.

(2) If you're wondering why we don't just use some variant of the Newton-Raphson method for training neural networks, instead of the clearly-dysfunctional gradient descent, the phrase "one instead has to left multiply the function $ F(x_n) $ by the inverse of its $ k \times k $ Jacobian matrix $ J_F(x_n) $" should be enough to scare you straight. Go read up on the Adam optimiser instead.

(3) I would include academic links here... but I'm not actually aware of any formal research on this relationship. One would think there'd be an equivalent of Kaiming-He for the learning rate? If anyone knows of such a paper, please fill me in.

(4) On "doubling all your inputs and actual outputs can effectively quadruple your learning rate": Consider the example of a simplified neural network without bias terms and using only ReLU, trained using quadratic loss. In this case, if we double both the input values (consequently doubling the predicted output value) and the actual output, then:

$$ \begin{split} w &\leftarrow w - \lambda \times \frac{dE'}{dw} \\ &= w - \lambda \times \frac{ d((out'-actual')^2) }{dw} \\ &= w - \lambda \times \frac{ d((2 \times out - 2 \times actual)^2) }{dw} \\ &= w - 4 \times \lambda \times \frac{ d((out - actual)^2) }{dw} \\ &= w - 4 \times \lambda \times \frac{dE}{dw} \end{split} $$

...Which we could rephrase as $ \lambda' = 4 \times \lambda $. This happens despite the weights essentially being dimensionless quantities which should thus not be affected by pure dimensional-scaling changes.