I'm a grad student in Mathematics, with little background in Machine Learning. I've recently come across the paper "A Disciplined Approach to Neural Network Hyper-Parameters : Part 1" by Leslie Smith, and I am really confused about his approach in Batch Size.
He proposes that when using the "1-Cycle Policy" to a model one should use larger batch sizes, contrary to earlier works saying that small batch sizes are preferable. (My small experimental models seems to support his idea too.) Many papers argues that using large batch sizes might be computationally inexpensive, but will lead to poor generalization, due to the fact that large batch size tends to converge to a sharp optimal point for unknown reasons.
In conclusion, I should use large batch sizes ONLY when I'm using CLR, and smaller batch sizes otherwise, right? But why? These two facts seems to contradict one another, and maybe it's because of my major or something, and it's hard for me to just "accept" this fact purely on the test results. I would be happy if someone could give me even a vague reason behind this!
Here's a link to the paper I'm referring : A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
[Edit] Here's a quote from "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima" (Nitish Shirish Keskar, et al.)
One natural avenue for improving parallelism is to increase the batch size |Bk|. This increases the amount of computation per iteration, which can be effectively distributed. However, practitioners have observed that this leads to a loss in generalization performance; see e.g. (LeCun et al., 2012). In other words, the performance of the model on testing data sets is often worse when trained with largebatch methods as compared to small-batch methods. In our experiments, we have found the drop in generalization (also called generalization gap) to be as high as 5% even for smaller networks.
and here's the link : On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima