Batch Normalization shift/scale parameters defeat the point

Question

According to the paper introducting Batch Normalization, the actual BN function is given as:

---

• Input: Values of $x$ over a mini-batch $\mathcal B = \{x_{1,\ldots,m}\}$; parameters to be learned $\gamma,\beta$.
• Output: $\{y_i = \mathrm{BN}_{\gamma,\beta}(x_i)\}$.

$\mu_{\mathcal B} \leftarrow \frac1m \sum_{i = 1}^m x_i$

$\sigma^2_{\mathcal B} \leftarrow \frac1m \sum_{i=1}^m (x_i - \mu_{\mathcal B})^2$

$\hat x_i \leftarrow \frac{x_i - \mu_{\mathcal B}}{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}$

$y_i \leftarrow \gamma \hat x_i + \beta \equiv \mathrm{BN}_{\gamma,\beta}(x_i)$

---

(Here, $\epsilon$ is some small constant added for numerical stability. The above is an almost exact copy of the box Algorithm 1, in section 3 of the paper linked above.)

Now, $\gamma,\beta$ are learned parameters, as far as I can tell on the level of each mini-batch. In particular, for a fixed mini-batch they can take any value. It seems to me that this makes shifting by the mean and scaling by the standard deviation pointless. The resulting output values are given by $$ y_i = \frac{\gamma}{{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}}x_i + \beta - \frac{\gamma }{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}\mu_{\mathcal B}. $$ Hence, if we define $$ \gamma' = \frac{\gamma}{{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}}, $$ $$ \beta' = \beta - \frac{\gamma }{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}\mu_{\mathcal B}, $$ we might as just have defined and learned values for $\beta',\gamma'$ and then returned $y_i = \gamma'x_i + \beta'$.

I presume that I misunderstand -- can someone explain where I went wrong?

Answer 1

I'm by no means an expert on this topic, but here are my thoughts.

I think the main point of batch normalization is that training of each network layer is unaffected by changes in scale of the preceding layers. The author wrote

$$BN(Wu) = BN((aW)u)$$

for any scalar $a$. Thus, also backpropagated error is unaffected as well. So there is no scale explosion, even when learning rate is high.

If you simply apply

$$BN(x) = y_i = \gamma_ix_i + \beta_i, \text{ where } x = Wu$$

the above no longer holds.

Parameters $\gamma$ and $\beta$ are a must have, because otherwise the normalized outputs $\hat{x_i}$ would be mostly close to $0$, which would hinder the network ability to fully utilize nonlinear transformations (authors give an example of sigmoid function, which is close to identity transformation in $0$ proximity)

Finally, one can argue that the input $y = Wx + b$ to the BN layer, as a combination of many factors, may resemble Gaussian distribution (CLT). So by studentizing it we keep the inputs stable and, hopefully, following standard normal distribution to some degree.

Answer 2

If someone passes by this question in the future, Ian Goodfellow and Yoshua Bengio and Aaron Courville's Deep Learning, Section 8.7.1 can give a good explanation:

Normalizing the mean and standard deviation of a unit can reduce the expressive power of the neural network containing that unit. To maintain the expressive power of the network, it is common to replace the batch of hidden unit activations $H$ with $\gamma H+\beta$ rather than simply the normalized $H$. The variables $\gamma$ and $\beta$ are learned parameters that allow the new variable to have any mean and standard deviation. At first glance, this may seem useless — why did we set the mean to $0$, and then introduce a parameter that allows it to be set back to any arbitrary value $\beta$?

The answer is that the new parametrization can represent the same family of functions of the input as the old parametrization, but the new parametrization has different learning dynamics. In the old parametrization, the mean of $H$ was determined by a complicated interaction between the parameters in the layers below $H$. In the new parametrization, the mean of $\gamma H+\beta$ is determined solely by $\beta$. The new parametrization is much easier to learn with gradient descent.

Answer 3

Correct me if I am wrong... The BN normalization values are calculated for each components to normalize the input to a neuron (i.e. elliptic distribution becomes circular). The resulting distribution may be too wide or too narrow and shifted to "activate" well the activation function. To my understanding, the gamma and beta learnt parameters are constants applied to all components (not gamma and beta values per component) learnt, re-scaling and shifting optimally the more circular distribution before applying the activation function. Thus not equivalent to only learn beta and gamma (which is already learnt by backprop in W and b coef) without prior normalization.