If the main goal of the learning rate is to decrease the cost function, why wouldn't it make sense to have a huge learning rate?

Since the formula would be

x <- x - n(f(x)) where n is the learning rate.

Is there something i am missing?


Learning rate is used to ensure convergence. A one line explanation against high learning rate would be:

The answer might overshoot the optimal point

There is a concept called momentum in neural network, which has almost the same application as that of the learning rate.

Initially, it would be better to explore more. So, a low momentum and high learning rate would be advisable.

Gradually, the momentum can be increased and the learning rate can be decreased for ensuring convergence.

  • $\begingroup$ Why would it overshoot the optimal point? I do not understand? Could you provide a simple example? How does gradient descent even ensure that cost is minimized? If it overshoots the optimal point, doesn't it mean that the cost increases? But the formula does not show a possibility of it $\endgroup$ – aceminer Nov 27 '15 at 4:27
  • $\begingroup$ @aceminer In addition to the video on the Math site, please have a look at this for better understanding $\endgroup$ – Dawny33 Nov 27 '15 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.