I have a simple gradient descent algorithm implemented in MATLAB which uses a simple momentum term to help get out of local minima.
% Update weights with momentum dw1 = alpha(n)*dJdW_1 + mtm*dw1; % input->hidden layer dw2 = alpha(n)*dJdW_2 + mtm*dw2; % hidden->output layer Wt1 = Wt1 - dw1; Wt2 = Wt2 - dw2;
I have been looking at implementing the Nesterov accelerated gradient descent method to improve this algorithm and have been following the tutorial here to do so.
% Update with Nesterov accelerated descent dw1_prev = dw1; dw2_prev = dw2; dw1 = alpha(n)*dJdW_1 - mtm*dw1; dw2 = alpha(n)*dJdW_2 - mtm*dw2; Wt1 = Wt1 - (1+mtm)*dw1 - mtm*dw1_prev; Wt2 = Wt2 - (1+mtm)*dw2 - mtm*dw2_prev;
However, this appears to converge more slowly than the simple momentum method. Could this be down to the data I am testing with or have I made a mistake in the implementation?