10
$\begingroup$

Is there any rule of thumb between depth of a neural network and learning rate? I have been noticing that the deeper the network is, the lower the learning rate must be.

If that's correct, why is that?

$\endgroup$
  • 1
    $\begingroup$ a related discussion for a boosting model. stats.stackexchange.com/questions/168666/… $\endgroup$ – Haitao Du Sep 2 '16 at 3:47
  • $\begingroup$ The discussion provides useful information but It doesn't answer my question. Could you please comment on it? $\endgroup$ – user_1177868 Sep 2 '16 at 12:20
  • 1
    $\begingroup$ ye, that's why i put it in comment but not answer, and upvoted your questions. $\endgroup$ – Haitao Du Sep 2 '16 at 12:45
4
$\begingroup$

This question has been answered here:

With neural networks, should the learning rate be in some way proportional to hidden layer sizes? Should they affect each other?

Short answer is yes, there is a relation. Though, the relation is not this trivial, all I can tell you that what you see is because the optimization surface becomes more complex as the the number of hidden layers increase, therefore smaller learning rates are generally better. While stucking in local minima is a possibility with low learning rate, it's much better than complex surface and high learning rate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.