If the main goal of the learning rate is to decrease the cost function, why wouldn't it make sense to have a huge learning rate?
Since the formula would be
x <- x - n(f(x)) where n is the learning rate.
Is there something i am missing?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Learning rate is used to ensure convergence. A one line explanation against high learning rate would be:
The answer might overshoot the optimal point
There is a concept called momentum in neural network, which has almost the same application as that of the learning rate.
Initially, it would be better to explore more. So, a low momentum and high learning rate would be advisable.
Gradually, the momentum can be increased and the learning rate can be decreased for ensuring convergence.
-
$\begingroup$ Why would it overshoot the optimal point? I do not understand? Could you provide a simple example? How does gradient descent even ensure that cost is minimized? If it overshoots the optimal point, doesn't it mean that the cost increases? But the formula does not show a possibility of it $\endgroup$– aceminerCommented Nov 27, 2015 at 4:27
-
$\begingroup$ @aceminer In addition to the video on the Math site, please have a look at this for better understanding $\endgroup$– Dawny33Commented Nov 27, 2015 at 5:55