Say the model is bouncing around the optimum with SGD. Is there a way to know that it's near the minimum, pause the model, and continue with an extremely small learning rate


That's one of the motivations of adaptive algorithms (e.g. Adam). In this algorithms the effective learning rate change during training and adapts the local geometry of the loss surface. You can take a look at this really nice post if you want to understand this more in depth.

I see that the title of your post exposes a different problem. When you take more samples to compute one gradient update, the variance of the estimate decreases, but at the same time the complexity of each update increases. Given that the stochastic estimates are unbiased, you can go much faster using minibatch or SGD.

Now to see if you are near a minimum in a very simplistic way, you could keep track of the magnitude of the gradient. When this value is close to zero for a certain number of iterations you may take more samples into account for each gradient update.

I cannot recall any paper that uses this approach but one similar way of using "better" updates when near the optimum is to use second order methods. This approach has been largely studied, since e.g. Newton's method has the fastest convergence rate when sufficiently close to the optimum, but each update is very expensive to compute.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.