# Do $\gamma$ and $\beta$ “undo” the eﬀects of batch normalization?

Let $$H$$ be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. The normalized version of $$H$$ is:

$$H' = (H − \mu) / \sigma$$

batch normalization reduces the expressiveness of a unit. To maintain the expressiveness, it is common to replace the batch of hidden unit activations not just with $$H'$$ but $$\gamma H' + \beta$$, where $$\gamma$$ and $$beta$$ are learned parameters which then adjust the hidden outputs to any mean and standard deviation.

Do $$\gamma$$ and $$\beta$$ “undo” the effects of batch normalization? Why?

• any ideas about this?! Mar 12, 2019 at 19:12
• no they don't undo Mar 14, 2019 at 2:56

The optimizer will train the $$\gamma, \beta$$ to minimize the error, so they only "undo" the scaling if this is the optimal thing to do.
Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity. To address this, we make sure that the transformation inserted in the network can represent the identity transform. To accomplish this, we introduce, for each activation $$x^{(k)}$$, a pair of parameters $$\gamma^{(k)}, \beta^{(k)}$$, which scale and shift the normalized value: $$y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}.$$ These parameters are learned along with the original model parameters, and restore the representation power of the network. Indeed, by setting $$\gamma^{(k)} = \sqrt{\text{Var}\left[x^{(k)}\right]}$$ and $$\beta^{(k)} = \mathbb{E}\left[x^{(k)}\right]$$, we could recover the original activations, if that were the optimal thing to do.