Nesterov accelerated gradient descent in neural networks

Question

I have a simple gradient descent algorithm implemented in MATLAB which uses a simple momentum term to help get out of local minima.

        % Update weights with momentum
        dw1 = alpha(n)*dJdW_1 + mtm*dw1; % input->hidden layer
        dw2 = alpha(n)*dJdW_2 + mtm*dw2; % hidden->output layer
        Wt1 = Wt1 - dw1;
        Wt2 = Wt2 - dw2;

I have been looking at implementing the Nesterov accelerated gradient descent method to improve this algorithm and have been following the tutorial here to do so.

        % Update with Nesterov accelerated descent
        dw1_prev = dw1;
        dw2_prev = dw2;
        dw1 = alpha(n)*dJdW_1 - mtm*dw1;
        dw2 = alpha(n)*dJdW_2 - mtm*dw2;
        Wt1 = Wt1 - (1+mtm)*dw1 - mtm*dw1_prev;
        Wt2 = Wt2 - (1+mtm)*dw2 - mtm*dw2_prev;

However, this appears to converge more slowly than the simple momentum method. Could this be down to the data I am testing with or have I made a mistake in the implementation?

Answer 1

I think that I have located the bug in my implementation. This method now does indeed converge faster and I believe the method to be correct for any future readers.

    % Update with Nesterov accelerated descent
    dw1_prev = dw1;
    dw2_prev = dw2;
    dw1 = alpha(n)*dJdW_1 + mtm*dw1;
    dw2 = alpha(n)*dJdW_2 + mtm*dw2;
    Wt1 = Wt1 - (1+mtm)*dw1 - mtm*dw1_prev;
    Wt2 = Wt2 - (1+mtm)*dw2 - mtm*dw2_prev;