# How to understand whether Stochastic Gradient Descent has converged?

I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot with the step sizes however I don't think they make that big of a difference. After running for 30 iterations, I don't think I have seen any convergence based on the gradient values. The norm of the gradient values change around 0.1 to 0.01. Not a constant decrease but it is erratic.

I am aware that it is not converging to optimal, I am okay with an approximate converge. I am just not really sure how to understand whether 0.1 norm(gradient) is close to the optimal and is relatively a good approximation or not. • 30 iterations or 30 epochs – shimao Dec 19 '18 at 23:49

• In fact, many algorithms have something like an early stopping criterion of the form 'if the loss function did not move by more than $\epsilon$ over the last $k$ epochs / iterations then stop' and $\epsilon$ and $k$ are in fact hyperparameters but with senseful standard values (i.e. $\epsilon = 0.00001$ and $k=10$ or so you should be good in the sense that I would rather play around with other hyperparameters of the model instead of the parameters of the optimization) – Fabian Werner Dec 20 '18 at 9:24