For deep learning, does the statement "batch size should be no more than 32" make sense from Yann LeCun? Does batch size over 100 is really meaningless, as I see some paper related to Deep Reinforcement Learning, there always have some batch size over 100.
In addition, if batch size should really be no more than 32, does it mean that high-performance GPU is useless?
 A: The source of the claim is a tweet and the paper which is being referred to: "Revisiting Small Batch Training for Deep Neural Networks" by
Dominic Masters, Carlo Luschi.
It's important to keep in mind that this is only a single preprint, and it's usually a good idea to be somewhat skeptical when reading such a bold claim. 
Moreover, the authors only tested the claim on image classification tasks, so the scope of the paper is limited. There has also been successful work with training using very very large batch sizes. 
The claim made is not nearly as strong as "any batch size above 32 will work poorly" or "any batch size above 32 is meaningless", but rather something much more subtle: larger batch sizes have a smaller range of hyperparameters for which training works well.
Also keep in mind that a batch size of 32 is quite sufficient in order to take advantage of computational parallelism on GPUs. In fact, many larger networks must be trained with a batch size of only 4 or 8 because of their size.
I do agree there is probably some truth in the claim that small batch sizes may work better, but the claim is much more limited than suggested, and doesn't suddenly invalidate all batch sizes larger than 32.
A: Without a context - I would say "no".
There are two aspects of batching:


*

*training/learning, e.g. mini-batch vs. online training - the size of the batch would depend on your implementation

*memory constraints - the size of the batch would depend on you hardware


So, what does 32 mean? Is it reasonable to limit your MNIST challenge batch to 32? Based on what? The memory should be non-issue with so small images. In terms of learning your algorithm's performance may vary depending on your batch size. So you should probably experiment with different batch-sizes, at least.
