Do $\gamma$ and $\beta$ “undo” the effects of batch normalization?

Question

Let $H$ be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. The normalized version of $H$ is:

$$H' = (H − \mu) / \sigma$$

batch normalization reduces the expressiveness of a unit. To maintain the expressiveness, it is common to replace the batch of hidden unit activations not just with $H'$ but $\gamma H' + \beta$, where $\gamma$ and $beta$ are learned parameters which then adjust the hidden outputs to any mean and standard deviation.

Do $\gamma$ and $\beta$ “undo” the effects of batch normalization? Why?

Answer 1

The optimizer will train the $\gamma, \beta$ to minimize the error, so they only "undo" the scaling if this is the optimal thing to do.

From the batch norm paper:

Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity. To address this, we make sure that the transformation inserted in the network can represent the identity transform. To accomplish this, we introduce, for each activation $x^{(k)}$, a pair of parameters $\gamma^{(k)}, \beta^{(k)}$, which scale and shift the normalized value: $$ y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}. $$ These parameters are learned along with the original model parameters, and restore the representation power of the network. Indeed, by setting $\gamma^{(k)} = \sqrt{\text{Var}\left[x^{(k)}\right]}$ and $\beta^{(k)} = \mathbb{E}\left[x^{(k)}\right]$, we could recover the original activations, if that were the optimal thing to do.

Emphasis mine.

"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." Sergey Ioffe, Christian Szegedy

Answer 2

No, they don't.

The purpose or effect of batch normalization of a given activation in a neural network, is essentially to reduce internal covariance shift of that activation. Arguably, this "internal covariance shift" is not explained well in the 2015 Batch Norm paper. Here I think it means, mathematically, distribution of input of that activation changes per sample batch, because of nonlinearity of previous layers this input comes from. As a result, gradient updates have to reflect that distribution change when it is the wrong the to do as it does not necessarily help move weights to the correct direction of reducing loss or optimization objectivity.

You already know it is beneficial to introduce scaling and shifting post-batch-normalization - to fully utilize nonliearity of network or to avoid 0 proximity in nonlinear function such as sigmoid. This scalar and shifter decouple the activation's distribution change from previous layers, thus reducing so-called covariance shift. Therefore, the activation is still "normalized" - its distribution is "fixed" thru the scaler and shifter. And although they are still subject to change as they are learned thru gradient descent as well during learning/training, you can probably see their changes are a lot slower and more steady compared to as if there's no batch-normalization and distribution depends on previous layers's nonlinearity.