What is the significance of a batch size of greater than one? Why do we use batch sizes in neural networks and deep learning methods such as ConvNets?
I mean rather than taking average of several steps and going in that direction which causes smoother movement in the parameter space, what are the benefits that we gain by doing this?
Does using batch-size=1 affect the performance badly? does using more batch-size translate to faster convergence?   
 A: It is recommend to read "Neural Networks: Tricks of the trade", all reason to improve parameter tuning are precisely describe in the book. And to your specific questions:
What are the benefits that we gain by doing this?


*

*As you said, most benefit from "smoother movement in parameter space"

*Parallel processing of multiple batches.


Does using batch-size=1 affect the performance badly?


*

*No, not as significantly as you think, just like gradient update without momentum term.


Does using more batch-size translate to faster convergence?


*

*More batch-size doesn't always lead to faster convergence, but instead, gradient update can be more stable.

A: *

*Performance usually drops with batch_size=1 if you supply your batch with python which usually 100+ slower than c code and preparing you batch usually done with batch operations (numpy)


2.From result standpoint the more batch size it is the closer it to real gradient (rather than stochastic) which can be desirable and faster coverage, but in neral networks usually a lot of local minimums so you want that randomness in gradient to get away from them. Thats why Limited-memory BFGS usually not work for NN.
