2
$\begingroup$

I'm a grad student in Mathematics, with little background in Machine Learning. I've recently come across the paper "A Disciplined Approach to Neural Network Hyper-Parameters : Part 1" by Leslie Smith, and I am really confused about his approach in Batch Size.

He proposes that when using the "1-Cycle Policy" to a model one should use larger batch sizes, contrary to earlier works saying that small batch sizes are preferable. (My small experimental models seems to support his idea too.) Many papers argues that using large batch sizes might be computationally inexpensive, but will lead to poor generalization, due to the fact that large batch size tends to converge to a sharp optimal point for unknown reasons.

In conclusion, I should use large batch sizes ONLY when I'm using CLR, and smaller batch sizes otherwise, right? But why? These two facts seems to contradict one another, and maybe it's because of my major or something, and it's hard for me to just "accept" this fact purely on the test results. I would be happy if someone could give me even a vague reason behind this!

Here's a link to the paper I'm referring : A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay

[Edit] Here's a quote from "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima" (Nitish Shirish Keskar, et al.)

One natural avenue for improving parallelism is to increase the batch size |Bk|. This increases the amount of computation per iteration, which can be effectively distributed. However, practitioners have observed that this leads to a loss in generalization performance; see e.g. (LeCun et al., 2012). In other words, the performance of the model on testing data sets is often worse when trained with largebatch methods as compared to small-batch methods. In our experiments, we have found the drop in generalization (also called generalization gap) to be as high as 5% even for smaller networks.

and here's the link : On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

$\endgroup$
  • $\begingroup$ Hi @shaine, welcome to CrossValidated! This is a nice question! What you call a contradiction is not very apparent to me from your formulation. Could you cite a paper arguing against Smith's claim? $\endgroup$ – Konstantin Nov 19 '19 at 17:04
  • $\begingroup$ Thanks for your attention, I'v edited my post! $\endgroup$ – shaine Nov 19 '19 at 17:21
  • 1
    $\begingroup$ Good batch size depends on your data, your model, and other hyperparameters (notably the learning rate). Use hyperparameter optimization to find the optimal for your problem. $\endgroup$ – jonnor Nov 19 '19 at 17:30
3
$\begingroup$

Indeed, there are many claims that large batch sizes do just as well, and just as many claims on the other side. Basically, it doesn't look like there's any consensus on the matter yet. For either side you can find as much theoretical justification as you'd like. (Obviously these aren't "actual" mathematical contradictions, since different authors start from different assumptions.)

These two facts seems to contradict one another, and maybe it's because of my major or something, and it's hard for me to just "accept" this fact purely on the test results.

I like to have theoretical justification as much as anyone, but neural networks is one place where the theory hasn't really caught up, and when the empirical evidence contradicts what theory suggests, it's probably a better idea to go with the evidence.


Priya Goyal et al. "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour" https://arxiv.org/abs/1706.02677

Elad Hoffer et al. "Train longer, generalize better: closing the generalization gap in large batch training of neural networks" https://arxiv.org/abs/1705.08741

Christopher J. Shallue et al. "Measuring the Effects of Data Parallelism on Neural Network Training." https://arxiv.org/abs/1811.03600

Nitish Shirish Keskar et al. "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima" https://arxiv.org/abs/1609.04836

Tao Lin et al. "Don't Use Large Mini-Batches, Use Local SGD" https://arxiv.org/abs/1808.07217

Dominic Masters and Carlo Luschi. "Revisiting Small Batch Training for Deep Neural Networks." https://arxiv.org/abs/1804.07612

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.