Let H be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. The normalized version of H is:

H' = (H − µ) / σ

batch normalization reduces the expressiveness of a unit. To maintain the expressiveness, it is common to replace the batch of hidden unit activations not just with H' but γ.H' + β, where γ and β are learned parameters which then adjust the hidden outputs to any mean and standard deviation.

Do γ and β “undo” the effects of batch normalization? (why?)

  • $\begingroup$ any ideas about this?! $\endgroup$ – hmojtaba Mar 12 at 19:12
  • $\begingroup$ no they don't undo $\endgroup$ – shimao Mar 14 at 2:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.