In the gradient descent method, the learning rate (which is multiplied by the results of the gradient on each weight) identifies the size of the step (steep down) that the algorithm takes in each iteration to reduce the objective cost function. How does it differ from the parameter momentum?
Thank you
Momentum is a whole different method, that uses parameter that works as an average of previous gradients.
Precisely in Gradient Descent (let's denote learning rate by $\eta$)
$$w_{i+1} = w_i - \eta \nabla F(w)$$
Whereas in Momentum Method
$$w_{i+1} = w_i - \gamma v_i$$
Where
$$v_{i+1} = \beta v_i + (1 - \beta) \nabla F(w)$$
Note that this method has two hyperparameters, instead of one like in GD, so I can't be sure if your momentum means $\gamma$ or $\beta$. If you use some software though, it should have two parameters.
Momentum is another parameter of SDG (as stated in the link) but you are saying it is not
$\endgroup$
– Katherine
Feb 18 '18 at 21:57
