0
$\begingroup$

I am really confused about applying gradient descent with momentum. The trusted resources which I use for learning about AI have different information. CS231n says to use momentum like this,
enter image description here

Same implementation is suggested by Michael Nielsen in his deep learning book. But Andrew Ng's deep learning course says this, enter image description here

What's happening? Are these two same, I tried to make sure that doesn't happen and I am pretty sure that they are not same. But enlighten me.

$\endgroup$
2
  • $\begingroup$ Does this answer your question? Can you explain what the variable are in the code/equations you provided? $\endgroup$ Commented Aug 15, 2019 at 7:58
  • $\begingroup$ @JanKukacka The formulations are not equivalent here (assuming $\alpha$ is the learning rate and $\beta$ the momentum term mu). I asked a similar question a few years back and I still don't have an idea whether there is a(n) (dis)advantage to using one formulation over the other. I assume that there are no practical differences... $\endgroup$ Commented Nov 15, 2019 at 10:47

1 Answer 1

0
$\begingroup$

Essentially the two version are not the same. In CS231 you have more degrees of freedom w.r.t the gradient and velocity terms, since their weights determined independently through alpha (lr) and beta, respectively. However, in NG version the weighting of lr and v is determined only by beta and after that alpha weights them both (by weighting the updated velocity term). Hence, I find CS231 preferable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.