2
$\begingroup$

I am currently trying to implement Nesterov Accelerated Gradient (NAG) in a neural network following the description shown here.

My understanding is that it is identical to 'Classical' momentum (CM), also shown in the link above, except that the gradients are calculated on weights += mu * v.

That seeming simple enough, I implemented this and to my surprise it seems that NAG doesn't always outperform CM (typically only a small improvement if any). In fact, I am often seeing CM outperform NAG! This seems wrong to me.

Is this expected behavior? I have looked at multiple other questions here on NAG and I'm pretty sure my code is sound.

Here is the relevant code (I know my calculate_gradients function works correctly, among other code excluded). The part I don't know for sure is everything outside the gradient calculations in the while loop. Have I perhaps missed something?

# use_nesterov = TRUE
# vt.old = 0
# mu = 0.9

while (epoch < stepmax){

    # update function
    vt = (mu * vt.old) - (learningrate * gradients)
    weights += vt

    vt.old = vt

    # other non-relevant code
    ...

    # calculate gradients
    gradients = calculate_gradients(
                    if (use_nesterov) {weights + mu*vt.old} else {weights}, 
                    ...)
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.