Im using gradient-descent-based algorithm for my problem where
new_value = old_value - Step_size*Gradient
For exit criteria, im determining the change in fn value between iteration i.e.,
if (old_Objective_fn_value - new_Objective_fn_value) <=0.001 exist otherwise continue.
For different Step_size, the algorithm meets the exit criteria at different point. For example, when my Step_size is x the final objective function value is p and when my Step_size is y the final objective function value is q.
I would like to know any logical reason why the algorithm converges at different objective fun values rather than at the same.
How can we make the algorithm converge to the same objective function value irrespective of the step size with the same exit criterion?