Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. Therefore there is no need to calculate the gradient for each example in the Training Set at each iteration, since there are examples that introduce little variance or have little impact on the construction of the full gradients at each iteration. The error of approximating gradients with stochastic gradients will vanish with many iterations.
So my question is the following:
What is the difference of providing a single training example at each iteration or a minibatch of examples? If we are talking about the unbiased estimator, then with a minibatch you will converge with fewer iterations, while using only 1 example per iteration, you will always converge to the same minimum point, but more slowly. Is what I am saying correct, or will a larger minibatch better approximate the full gradients at each iteration, and thus have better performance?
(Not taking into account that for very complex deep learning models, a smaller minibatch allows to perform a regularization effect, so the adaptation to noise allows to generalize better)
