# Hinton claims SGD with batch norm can help: How?

In Hinton's paper "Layer Normalization", on the first page he says

Feedforward neural networks trained using batch normalization converge faster even with simple SGD.

By this I think he means do batch normalization with batch size one. So the variance (across the batch) is always zero, and the mean (across the batch) is always just the value of the unit.

Suppose I have a neural network with a single hidden layer which I train with batch normalization of size one. The batch normalization layer proceeds by first subtracting the mean and dividing by the variance. (But this would just zero out all the units if my batch is of size one.) Then it learns shift/scale parameters $\beta$ and $\gamma$ for each unit, and if $x$ is zero, of course $\gamma x + \beta = \beta$.

So it seems like using batch normalization with batch size one ends up just training some constant function $f(\boldsymbol{\beta})$, where $f$ is the nonlinearity.

What am I missing here? How could strict SGD with batch normalization be useful?

• Heh? Why do you think Hinton is talking about batches of size 1? Mar 11, 2018 at 22:13
• @amoeba Glad to know that my mistake is that simple. When he says "simple" SGD, I assumed he was contrasting with minibatch GD. But as I read it now, I bet he is contrasting with things like momentum, adagrad, other fancier methods. Do you agree? // I think my understanding of the term "stochastic gradient descent" is too narrow--I thought it always implied batch size one. Mar 11, 2018 at 22:35
• Yes, I agree. The term "SGD", how it is currently used in practice, definitely does not imply batch size one. In most cases it refers to minibatch GD. Mar 11, 2018 at 22:57
• @amoeba Do you recommend deleting the question, or answering it myself? Mar 11, 2018 at 23:21