As far as I understand it, batch norm normalises all the input features to a layer to a unit normal distribution, $\mathcal{N}(\mu=0,\sigma=1)$. The mean and variance $\mu, \sigma^2$ are estimated by measuring their values for the current mini-batch.

After the normalisation the inputs are scaled and shifted by scalar values:

$$\hat{x}_i' = \gamma \hat{x}_i + \beta$$

(Correct me if I'm wrong here - this is where I start to get a bit unsure.)

$\gamma$ and $\beta$ are scalar values and there is a pair of each for every batch-normed layer. They are learnt along with the weights using backprop and SGD.

My question is, aren't these parameters redundant because the inputs can be scaled and shifted in any way by the weights in the layer itself. In other words, if

$$y = W \hat{x}' + b$$


$$\hat{x}' = \gamma \hat{x} + \beta$$


$$y = W' \hat{x} + b'$$

where $W' = W\gamma$ and $b'=W\beta + b$.

So what is the point of adding them of the network is already capable of learning the scale and shift? Or am I totally misunderstanding things?


1 Answer 1


There is a perfect answer in the Deep Learning Book, Section 8.7.1:

Normalizing the mean and standard deviation of a unit can reduce the expressive power of the neural network containing that unit. To maintain the expressive power of the network, it is common to replace the batch of hidden unit activations H with γH+β rather than simply the normalized H. The variables γ and β are learned parameters that allow the new variable to have any mean and standard deviation. At first glance, this may seem useless — why did we set the mean to 0, and then introduce a parameter that allows it to be set back to any arbitrary value β?

The answer is that the new parametrization can represent the same family of functions of the input as the old parametrization, but the new parametrization has different learning dynamics. In the old parametrization, the mean of H was determined by a complicated interaction between the parameters in the layers below H. In the new parametrization, the mean of γH+β is determined solely by β. The new parametrization is much easier to learn with gradient descent.

  • 2
    $\begingroup$ Still the answer is too vague. And is missing depth. What does better learning dynamic means? Why does SGD works better when the mean is determined only by beta vs when it is determined by previous layers weights and the inputs? I hear all kind of arguments around the shape of the loss function (and how it can be ill conditioned) without batch normalization, but I don’t know if anyone really managed to prove that or measure that experimentally $\endgroup$
    – user25322
    Sep 30, 2020 at 19:21
  • 1
    $\begingroup$ @user25322 papers.nips.cc/paper/2018/file/… this paper is exactly about it. $\endgroup$ Dec 4, 2020 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.