1
$\begingroup$

Problem: Im unsure if I understood Nesterov Optimization

Im writing about Nesterov Optimization, but the notation im using seems different from the references below. I have done it using some books as guides.

Would someone please clarify?


Let $\epsilon$ be the learning rate, $w$ each weight of the neural network, $\alpha$ the momentum and $E$ a loss function and considering the weights and gradients are calcualted as an unidimensional vector, the weight updates is done as below :

$n_0 = 0 $

$n_t = \alpha * n_{t-1} + \epsilon \frac{\partial E}{\partial w_t}$

And the update for each weight done as the formula below:

$\Delta_{w(t)} = \alpha_{n{t-1}} - {1 - \alpha} n_t$


QUESTIONS

What exactly is $n$ and $t$ ?


References:

https://blogs.princeton.edu/imabandit/2014/03/06/nesterovs-accelerated-gradient-descent-for-smooth-and-strongly-convex-optimization/

http://ruder.io/optimizing-gradient-descent/

What's the difference between momentum based gradient descent, and Nesterov's accelerated gradient descent?

http://neuralnetworksanddeeplearning.com/chap2.html

https://brilliant.org/wiki/backpropagation/

$\endgroup$
2
+100
$\begingroup$

I think you might be missing something here. I will use the first link you provided as a guide.

Nesterov's method has two steps: The normal gradient update, and then the "nudge" where we move the update a bit according to the update in the previous time step.

From what you write $n$ should be the change in your weights, i.e. is a vector. $t$ is the iteration number. So it doesn't make sense to initialize it to zero, the momentum is what we initialize to 0. Adapting notation from Bubeck's article, and changing the definition of $n$:

$$ n_0 = 0, n_t = \frac{1 + \sqrt{1+4n_{t-1}^2}}{2}, \alpha_t=\frac{1-n_t}{n_{t+1}} $$

Then the update is done in two steps: First get the regular update according to the gradient, previous weights and learning rate, then apply momentum to it:

$$ y_{t+1} = w_t - \epsilon \frac{\partial E}{\partial w_t}\\ w_{t+1} = (1 - \alpha_t)y_{t+1} + \alpha_t y_t $$

$\endgroup$
  • $\begingroup$ Awesome, would you just please clarify what is $alpha_t$ and $y_{t+1}$ and $w_{t+1}$. ? $\endgroup$ – KenobiShan Oct 26 '17 at 16:27
  • $\begingroup$ $\alpha_t$ is the momentum, $y_{t+1}$ is the intermediate weights (before applying momentum) and $w_{t+1}$ are the updated weights, after momentum has been applied. $\endgroup$ – Bar Oct 26 '17 at 16:55
  • $\begingroup$ Alright, just a last question, $n$ is not clear. You said its the change in the weights, but isnt that defined already by $w_{t+1}$ ? $\endgroup$ – KenobiShan Oct 26 '17 at 17:26
  • $\begingroup$ So wait... at the first formula, $n$ should be $\alpha$ ? It's okay if we need to change the letters. Help plx $\endgroup$ – KenobiShan Oct 26 '17 at 17:33
  • $\begingroup$ $n$ and $\alpha$ are what you define them to be. I chose to use $n$ as you were using it as well. They you defined it in the question wouldn't make sense because you initialize it to a scalar, 0, and the assign a vector to it (derivative). Unless you make the assumption that $w$ is a scalar as well, this does not hold. $\endgroup$ – Bar Oct 27 '17 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.