# Batch Normalization shift/scale parameters defeat the point

According to the paper introducting Batch Normalization, the actual BN function is given as:

• Input: Values of $x$ over a mini-batch $\mathcal B = \{x_{1,\ldots,m}\}$; parameters to be learned $\gamma,\beta$.
• Output: $\{y_i = \mathrm{BN}_{\gamma,\beta}(x_i)\}$.

$\mu_{\mathcal B} \leftarrow \frac1m \sum_{i = 1}^m x_i$

$\sigma^2_{\mathcal B} \leftarrow \frac1m \sum_{i=1}^m (x_i - \mu_{\mathcal B})^2$

$\hat x_i \leftarrow \frac{x_i - \mu_{\mathcal B}}{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}$

$y_i \leftarrow \gamma \hat x_i + \beta \equiv \mathrm{BN}_{\gamma,\beta}(x_i)$

(Here, $\epsilon$ is some small constant added for numerical stability. The above is an almost exact copy of the box Algorithm 1, in section 3 of the paper linked above.)

Now, $\gamma,\beta$ are learned parameters, as far as I can tell on the level of each mini-batch. In particular, for a fixed mini-batch they can take any value. It seems to me that this makes shifting by the mean and scaling by the standard deviation pointless. The resulting output values are given by $$y_i = \frac{\gamma}{{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}}x_i + \beta - \frac{\gamma }{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}\mu_{\mathcal B}.$$ Hence, if we define $$\gamma' = \frac{\gamma}{{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}},$$ $$\beta' = \beta - \frac{\gamma }{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}\mu_{\mathcal B},$$ we might as just have defined and learned values for $\beta',\gamma'$ and then returned $y_i = \gamma'x_i + \beta'$.

I presume that I misunderstand -- can someone explain where I went wrong?

• I had the same question and found the answer here. TL;DR the explicit mean and variance values are redundant but they make it easier to train. – Timmmm Jan 16 '19 at 17:00

I'm by no means an expert on this topic, but here are my thoughts.

I think the main point of batch normalization is that training of each network layer is unaffected by changes in scale of the preceding layers. The author wrote

$$BN(Wu) = BN((aW)u)$$

for any scalar $a$. Thus, also backpropagated error is unaffected as well. So there is no scale explosion, even when learning rate is high.

If you simply apply

$$BN(x) = y_i = \gamma_ix_i + \beta_i, \text{ where } x = Wu$$

the above no longer holds.

Parameters $\gamma$ and $\beta$ are a must have, because otherwise the normalized outputs $\hat{x_i}$ would be mostly close to $0$, which would hinder the network ability to fully utilize nonlinear transformations (authors give an example of sigmoid function, which is close to identity transformation in $0$ proximity)

Finally, one can argue that the input $y = Wx + b$ to the BN layer, as a combination of many factors, may resemble Gaussian distribution (CLT). So by studentizing it we keep the inputs stable and, hopefully, following standard normal distribution to some degree.

• Unless I misunderstand, while it is true that $\mathrm{BN}(Wu) = \mathrm{BN}((aW)u)$, the function actually depends on two more arguments, which are hidden here. A more explicit equation would be $\mathrm{BN}_{\beta,\gamma}(Wu) = \mathrm{BN}_{\beta,\gamma}((aW)u)$. And that is all nice and true, but if $\beta,\gamma$ themselves may vary depending on which input we are talking about, the equation does not really tell us anything. I presume this is where my misunderstanding arises, but: I do still misunderstand. – Mees de Vries Apr 5 '17 at 20:17
• @MeesdeVries Why do you think it doesn't tell us anything, what does it matter that $\beta_i$ and $\gamma_i$ vary for each $i$. – Łukasz Grad Apr 5 '17 at 20:35
• @MeesdeVries $\gamma$ and $\beta$ are learned parameters, just like any others in network - weights and biases. You may even throw away bias nodes because $\beta$ acts as bias, and they don't change from batch to batch (maybe this is were you're stuck) you simply update them every batch according to batch gradient – Łukasz Grad Apr 5 '17 at 20:42
• Ah, I see it now. Earlier on the page, $\gamma,\beta$ are introduced with an index (as $\gamma^{(k)}, \beta^{(k)}$, leading me in my haste to erroneously read that the values varied of $\beta,\gamma$ varied with the input; but these are just the coordinates of the (single, trained) vectors $\gamma,\beta$. Thank you! – Mees de Vries Apr 6 '17 at 8:41

If someone passes by this question in the future, Ian Goodfellow and Yoshua Bengio and Aaron Courville's Deep Learning, Section 8.7.1 can give a good explanation:

Normalizing the mean and standard deviation of a unit can reduce the expressive power of the neural network containing that unit. To maintain the expressive power of the network, it is common to replace the batch of hidden unit activations $$H$$ with $$\gamma H+\beta$$ rather than simply the normalized $$H$$. The variables $$\gamma$$ and $$\beta$$ are learned parameters that allow the new variable to have any mean and standard deviation. At first glance, this may seem useless — why did we set the mean to $$0$$, and then introduce a parameter that allows it to be set back to any arbitrary value $$\beta$$?

The answer is that the new parametrization can represent the same family of functions of the input as the old parametrization, but the new parametrization has different learning dynamics. In the old parametrization, the mean of $$H$$ was determined by a complicated interaction between the parameters in the layers below $$H$$. In the new parametrization, the mean of $$\gamma H+\beta$$ is determined solely by $$\beta$$. The new parametrization is much easier to learn with gradient descent.

Correct me if I am wrong... The BN normalization values are calculated for each components to normalize the input to a neuron (i.e. elliptic distribution becomes circular). The resulting distribution may be too wide or too narrow and shifted to "activate" well the activation function. To my understanding, the gamma and beta learnt parameters are constants applied to all components (not gamma and beta values per component) learnt, re-scaling and shifting optimally the more circular distribution before applying the activation function. Thus not equivalent to only learn beta and gamma (which is already learnt by backprop in W and b coef) without prior normalization.