Does using random minibatchs give more resilance against local minima, vs full batch gradient descent

Question

It was my believe that one of the advantages of using minibatches, when training a neural network via gradient descent (be it "vanilla" or the latest flavour of AdaGrad), was an increased resiliance against local minima.

I thought I read it somewhere, but when challanged I find that I can not produce a reference for it. I expected to find it in Y. A. LeCun and L. Bottou, “Efficient backprop,” in Neural networks: Tricks of the trade, Springer, 2012, pp. 9–48. but it was not there.

It logically makes sense to me that this would be the case. Minibatchs add noise to the training process, so local minima for one minibatch would be bounced out of for a different one. And since you generate a new set of random minibatches every epoch, that will mean not seeing the same local minima ever again. But that isn't very strong, since those local minima for minibatchs could in the scheme of things all be associated with the same full batch local minima.

Anyway, I would like a reference for or against my belief.

Answer 1

You have not done a very good job at reading

Y. A. LeCun and L. Bottou, “Efficient backprop,”
in Neural networks: Tricks of the trade, Springer, 2012, pp. 9–48.

it is in Section 4.1

Stochastic learning also often results in better solutions because of the noise in the updates. Nonlinear networks usually have multiple local minima of differ- ing depths. The goal of training is to locate one of these minima. Batch learning will discover the minimum of whatever basin the weights are initially placed. In stochastic learning, the noise present in the updates can result in the weights jumping into the basin of another, possibly deeper, local minimum. This has been demonstrated in certain simplified cases [15,30]

...

[15] T.M. Heskes and B. Kappen. On-line learning processes in artificial neural net- works. In J. G. Tayler, editor, Mathematical Approaches to Neural Networks, vol- ume 51, pages 199-233. Elsevier, Amsterdam, 1993.

[30] G. B. Orr. Dynamics and Algorithms for Stochastic learning. PhD thesis, Oregon Graduate Institute, 1995.

Note that earlier in this section LeCun and Bottou define stochastic as what you call a minibatch (giving example of using samples of 100 taken from 1000), and batch as what you call full batch