I have a simple gradient descent algorithm implemented in MATLAB which uses a simple momentum term to help get out of local minima.
% Update weights with momentum
dw1 = alpha(n)*dJdW_1 + mtm*dw1; % input->hidden layer
dw2 = alpha(n)*dJdW_2 + mtm*dw2; % hidden->output layer
Wt1 = Wt1 - dw1;
Wt2 = Wt2 - dw2;
I have been looking at implementing the Nesterov accelerated gradient descent method to improve this algorithm and have been following the tutorial here to do so.
% Update with Nesterov accelerated descent
dw1_prev = dw1;
dw2_prev = dw2;
dw1 = alpha(n)*dJdW_1 - mtm*dw1;
dw2 = alpha(n)*dJdW_2 - mtm*dw2;
Wt1 = Wt1 - (1+mtm)*dw1 - mtm*dw1_prev;
Wt2 = Wt2 - (1+mtm)*dw2 - mtm*dw2_prev;
However, this appears to converge more slowly than the simple momentum method. Could this be down to the data I am testing with or have I made a mistake in the implementation?