Why is batch normalization preserving the capacity of a network?

Question

I have a question regarding the Batch normalization paper of Sergey Ioffe

In the paper the author states on page 3 after discussion of derivation:

..., BN transform is a differentiable transformation that introduces normalized activations into the network. This ensures that as the model is training, layers can continue learning on input distributions that exhibit less internal covariate shift, thus accelerating the training. Furthermore, the learned affine transform applied to these normalized activations allows the BN transform to represent the identity transformation and preserves the network capacity

Question Why is this at all related to preserving the network capacity?

Edit: I read up a little now and think as the normalization is just like a copy process that does not change the number of parameters the author just makes this statement to ensure the benefit of this method. I first thought there is some twist here that makes the notion of capacity important.

Answer 1

To get a sense for why batch normalization is important, consider a neural network with two hidden layers, $h^1$ and $h^2$, and inputs $x$, so that the input to $h^2$ depends on the activation of $h^1$. However, during SGD, the parameters of both hidden layers are changing , so that the distribution of the activations of $h^1$ are constantly changing (different mean, different standard deviation, etc). This makes it difficult to accurately optimize the weights of $h^2$, since the weights are having to continuously react and update to the new distribution of inputs from the activations of $h^1$.

In this way, batch normalization aims to reduce this so-called "covariance shift" in the inputs to $h^2$, so that training can proceed smoother. They do this by standardizing each layer's activations. Let $a^1$ denote the activations of the first hidden layer. The standardized versions, $\hat{a}^1$ are given by:

$$\hat{a}^1 = \frac{a^1 - \overline{a}^1}{\text{SD}(a^1)}$$

Where $\overline{a}$ denotes the empirical mean, and SD denotes the standard deviation. In this way, we ensure that the inputs to $h^2$ will always have mean 0 and standard deviation 1.

However, this process of standardization throws away a lot of information about the previous layer, which may lead to a decreased ability for the model to learn complicated interactions - this is the notion of capacity you were referring to. BatchNorm resolves this by adding two parameters, $\gamma$ and $\beta$, again trained by backpropogation, that allows us to "add back" some information of the previous layer:

$$y^1 = \gamma\hat{a}^1 + \beta$$

The $y^1$'s are now used as input for $h^2$. Note that when $\gamma = \text{SD}(a)$, and $\beta = \overline{a}$, we recover the original inputs, which is the identity transform the original paper was referring to. And, when $\gamma = 1, \; \beta = 0$, we again have our completely standardized inputs. In this way, we allow for a balance between reducing the effects of covariance shift, while simultaneously maintaining the expressiveness of our neural network.