In Hinton's paper "Layer Normalization", on the first page he says

Feedforward neural networks trained using batch normalization converge faster even with simple SGD.

By this I think he means do batch normalization with batch size one. So the variance (across the batch) is always zero, and the mean (across the batch) is always just the value of the unit.

Suppose I have a neural network with a single hidden layer which I train with batch normalization of size one.

enter image description here

The batch normalization layer proceeds by first subtracting the mean and dividing by the variance. (But this would just zero out all the units if my batch is of size one.) Then it learns shift/scale parameters $\beta$ and $\gamma$ for each unit, and if $x$ is zero, of course $\gamma x + \beta = \beta$.

So it seems like using batch normalization with batch size one ends up just training some constant function $f(\boldsymbol{\beta})$, where $f$ is the nonlinearity.

What am I missing here? How could strict SGD with batch normalization be useful?

  • 1
    $\begingroup$ Heh? Why do you think Hinton is talking about batches of size 1? $\endgroup$
    – amoeba
    Mar 11, 2018 at 22:13
  • 1
    $\begingroup$ @amoeba Glad to know that my mistake is that simple. When he says "simple" SGD, I assumed he was contrasting with minibatch GD. But as I read it now, I bet he is contrasting with things like momentum, adagrad, other fancier methods. Do you agree? // I think my understanding of the term "stochastic gradient descent" is too narrow--I thought it always implied batch size one. $\endgroup$
    – Eric Auld
    Mar 11, 2018 at 22:35
  • 1
    $\begingroup$ Yes, I agree. The term "SGD", how it is currently used in practice, definitely does not imply batch size one. In most cases it refers to minibatch GD. $\endgroup$
    – amoeba
    Mar 11, 2018 at 22:57
  • $\begingroup$ @amoeba Do you recommend deleting the question, or answering it myself? $\endgroup$
    – Eric Auld
    Mar 11, 2018 at 23:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.