2
$\begingroup$

In the gradient descent method, the learning rate (which is multiplied by the results of the gradient on each weight) identifies the size of the step (steep down) that the algorithm takes in each iteration to reduce the objective cost function. How does it differ from the parameter momentum?

Thank you

| cite | improve this question | |
$\endgroup$
2
$\begingroup$

Momentum is a whole different method, that uses parameter that works as an average of previous gradients.

Precisely in Gradient Descent (let's denote learning rate by $\eta$)

$$w_{i+1} = w_i - \eta \nabla F(w)$$

Whereas in Momentum Method

$$w_{i+1} = w_i - \gamma v_i$$

Where

$$v_{i+1} = \beta v_i + (1 - \beta) \nabla F(w)$$

Note that this method has two hyperparameters, instead of one like in GD, so I can't be sure if your momentum means $\gamma$ or $\beta$. If you use some software though, it should have two parameters.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much for the clear explanation. I am using (stochastic gradient descent SGD) implementation by scikit-learn here scikit-learn.org/stable/modules/generated/…. I thought Momentum is another parameter of SDG (as stated in the link) but you are saying it is not $\endgroup$ – Katherine Feb 18 '18 at 21:57
  • $\begingroup$ Shouldn't you swap out $\gamma$ with $\eta$? They are the same thing and you are confusing OP. $\endgroup$ – generic_user Feb 18 '18 at 21:59
  • $\begingroup$ @generic_user I wanted to emphasise that these are two different methods. I also think this way is actually less confusing. $\endgroup$ – Jakub Bartczuk Feb 18 '18 at 22:06
  • 1
    $\begingroup$ They're really not. Momentum is just gradient descent with an extra feature. $\endgroup$ – generic_user Feb 18 '18 at 22:07
  • $\begingroup$ @generic_user well that's an opinion. They are different in that they update the weights with different values. Also, don't you think that this discussion is more confusing than the parameter naming issue? ;) $\endgroup$ – Jakub Bartczuk Feb 18 '18 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.