In Hinton's paper "Layer Normalization", on the first page he says
Feedforward neural networks trained using batch normalization converge faster even with simple SGD.
By this I think he means do batch normalization with batch size one. So the variance (across the batch) is always zero, and the mean (across the batch) is always just the value of the unit.
Suppose I have a neural network with a single hidden layer which I train with batch normalization of size one.
The batch normalization layer proceeds by first subtracting the mean and dividing by the variance. (But this would just zero out all the units if my batch is of size one.) Then it learns shift/scale parameters $\beta$ and $\gamma$ for each unit, and if $x$ is zero, of course $\gamma x + \beta = \beta$.
So it seems like using batch normalization with batch size one ends up just training some constant function $f(\boldsymbol{\beta})$, where $f$ is the nonlinearity.
What am I missing here? How could strict SGD with batch normalization be useful?
