I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot with the step sizes however I don't think they make that big of a difference. After running for 30 iterations, I don't think I have seen any convergence based on the gradient values. The norm of the gradient values change around 0.1 to 0.01. Not a constant decrease but it is erratic.

I am aware that it is not converging to optimal, I am okay with an approximate converge. I am just not really sure how to understand whether 0.1 norm(gradient) is close to the optimal and is relatively a good approximation or not. enter image description here

  • $\begingroup$ 30 iterations or 30 epochs $\endgroup$ – shimao Dec 19 '18 at 23:49

I am assuming by 30 iterations you mean 30 epochs. And every epoch has a number of iterations. If that is the case it would help you to plot the loss function after every epoch and check it's behaviour.

If it falls for a while and then keeps on rising and falling it is possible that it has found a local minima and is stuck around that.

If the trend shows a steady fall throughout and does not change after increasing the number of epochs/iterations it is possible that it is near a global minima.

A visual representation of the loss will help you in understanding how your SGD is behaving.

  • 1
    $\begingroup$ In fact, many algorithms have something like an early stopping criterion of the form 'if the loss function did not move by more than $\epsilon$ over the last $k$ epochs / iterations then stop' and $\epsilon$ and $k$ are in fact hyperparameters but with senseful standard values (i.e. $\epsilon = 0.00001$ and $k=10$ or so you should be good in the sense that I would rather play around with other hyperparameters of the model instead of the parameters of the optimization) $\endgroup$ – Fabian Werner Dec 20 '18 at 9:24
  • $\begingroup$ @Saksham Thanks for your reply! Yes I mean 30 epochs, and basically at each epoch I am using each example in the training set once, so the number of iterations is going to be equal to the size of the training set. I ran for 50 epochs recently, and added the visual representation of the loss function to the question - which is basically MSE. However I couldn't make any sense out of it. I am slowly decreasing alpha to get a convergence, but that also doesn't seem to be working. Maybe I should increase the number of epochs? $\endgroup$ – Aybike Dec 20 '18 at 16:00
  • $\begingroup$ @Aybike I do see a trend a gradual downward trend in the MSE. You should try increasing the number of epochs and that trend might be more visible. Sometimes, the loss function gets stuck at a local minima and we need to increase the learning rate to get it out of there. I would suggest you to experiment with your learning rate and try out 'cyclical learning rates' to encounter that. Also what is the model that you are using? If it is a neural network, adding more layers and making the function more complex might give better decrease in MSE with SGD. $\endgroup$ – Saksham Malhotra Dec 22 '18 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.