Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/abs/1609.04836

mentions that "large-batch methods are almost invariably attracted to regions with sharp minima and that, unlike small batch methods, are unable to escape basins of these minimizers". I guess they mean local minima with steep gradients around them when they say "sharp minima". Is this correct?

If small batch methods can escape them, are using small batch methods then similar in effect than using simulated annealing? In annealing one ads some random jumps into the optimization path. Using not all the data, but some of it in the small batch gradient descent or stochastic gradient descend (meaning a batch contains only one data sample), there is some random element where the optimization path "jumps" to - the path is not that smooth anymore as seen in the graphic of https://stats.stackexchange.com/a/153535

So, is mini-batch / stochastic gradient descend similar implicitly adding the same effect as simulated annealing?

Further read: Tradeoff batch size vs. number of iterations to train a neural network


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.