1
$\begingroup$

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. Therefore there is no need to calculate the gradient for each example in the Training Set at each iteration, since there are examples that introduce little variance or have little impact on the construction of the full gradients at each iteration. The error of approximating gradients with stochastic gradients will vanish with many iterations.

So my question is the following:

What is the difference of providing a single training example at each iteration or a minibatch of examples? If we are talking about the unbiased estimator, then with a minibatch you will converge with fewer iterations, while using only 1 example per iteration, you will always converge to the same minimum point, but more slowly. Is what I am saying correct, or will a larger minibatch better approximate the full gradients at each iteration, and thus have better performance?

(Not taking into account that for very complex deep learning models, a smaller minibatch allows to perform a regularization effect, so the adaptation to noise allows to generalize better)

$\endgroup$

1 Answer 1

0
$\begingroup$

...while using only 1 example per iteration, you will always converge to the same minimum point, but more slowly.

Typically, yes, but not always. Imagine a scenario where your individual samples pull you to a vicinity of another local minimum. Then, it's highly likely that you may not end up in the same local minimum as the mini-batch/batch updates lead to.

Variance typically slows the algorithm down, but it also increases the chance of going out of the course, which sometimes lead to better results.

If there is only one local optimum, both updates will converge to it via suitable learning rates.

$\endgroup$
1
  • $\begingroup$ I understand, thank you very much, but why it is said that it allows to have a non biased estimation of gradients at each iteration, maybe it is not proved or it is just an empirical result? Or does it mean that it is NOT a consistent estimator? So the size of the mini-batch turns out to be a hyper-parameter? $\endgroup$ Feb 2, 2021 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.