I have implemented NAG following this tutorial


It works, in fact with mu = 0.95 I get a good speed-up in learning compared to standard gradient descent, but I am not sure I implemented it correctly. I have a doubt about the gradient estimate I am using. This is the matlab code:

% backup previous velocity
vb_prev = nn.vb{l};
vW_prev = nn.vW{l};

% update velocity with adapted learning rates
nn.vb{l} = nn.mu*vb_prev - lrb.*nn.db{l};
nn.vW{l} = nn.mu*vW_prev - lrW.*nn.dW{l};

%update weights and biases 
nn.b{l} = nn.b{l} - nn.mu*vb_prev + (1 + nn.mu)*nn.vb{l};
nn.W{l} = nn.W{l} - nn.mu*vW_prev + (1 + nn.mu)*nn.vW{l};

which is exactly what is written in the tutorial. db and dW are my gradient estimates, but I am not sure if they are correct since I estimate them at W, not at W_ahead = W + mu*v.

Should I change something before forward and back-propagation to have a different gradient estimate? Or is NAG correct like this?

It works, but I would like to have a correct implementation of NAG.

Any help is appreciated, thank you very much!

  • $\begingroup$ does this answer help? $\endgroup$ – dontloo Aug 2 '16 at 9:56
  • $\begingroup$ Ouch sorry I flagged the answer by mistake... How can I remove the flag? $\endgroup$ – Steve3nto Aug 2 '16 at 10:57
  • 2
    $\begingroup$ lol i dont know, maybe you could add some explanation under that answer so hopefully the mediator will see it? $\endgroup$ – dontloo Aug 2 '16 at 11:02
  • 1
    $\begingroup$ @Steve3nto: I don't think you can retract flags on comments. (On questions/answers you've already flagged, the flag link should give you the option to retract the flag.) Don't worry about it - I've dismissed the flag. $\endgroup$ – Scortchi Aug 2 '16 at 11:41
  • $\begingroup$ Ok thanks. I think the other answer helps, it seems applecider had a very similar implementation as mine and according to dontloo it should achievethe correct NAG update. But I don't fully understand how it can work properly if the gradient is computed always at W and not at W_ahead = W + mu*v. $\endgroup$ – Steve3nto Aug 2 '16 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.