1
$\begingroup$

Let $H$ be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. The normalized version of $H$ is:

$$H' = (H − \mu) / \sigma$$

batch normalization reduces the expressiveness of a unit. To maintain the expressiveness, it is common to replace the batch of hidden unit activations not just with $H'$ but $\gamma H' + \beta$, where $\gamma$ and $beta$ are learned parameters which then adjust the hidden outputs to any mean and standard deviation.

Do $\gamma$ and $\beta$ “undo” the effects of batch normalization? Why?

$\endgroup$
2
  • $\begingroup$ any ideas about this?! $\endgroup$
    – hmojtaba
    Mar 12, 2019 at 19:12
  • $\begingroup$ no they don't undo $\endgroup$
    – shimao
    Mar 14, 2019 at 2:56

1 Answer 1

2
$\begingroup$

The optimizer will train the $\gamma, \beta$ to minimize the error, so they only "undo" the scaling if this is the optimal thing to do.

From the batch norm paper:

Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity. To address this, we make sure that the transformation inserted in the network can represent the identity transform. To accomplish this, we introduce, for each activation $x^{(k)}$, a pair of parameters $\gamma^{(k)}, \beta^{(k)}$, which scale and shift the normalized value: $$ y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}. $$ These parameters are learned along with the original model parameters, and restore the representation power of the network. Indeed, by setting $\gamma^{(k)} = \sqrt{\text{Var}\left[x^{(k)}\right]}$ and $\beta^{(k)} = \mathbb{E}\left[x^{(k)}\right]$, we could recover the original activations, if that were the optimal thing to do.

Emphasis mine.

"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." Sergey Ioffe, Christian Szegedy

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.