Let H be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. The normalized version of H is:
H' = (H − µ) / σ
batch normalization reduces the expressiveness of a unit. To maintain the expressiveness, it is common to replace the batch of hidden unit activations not just with H' but γ.H' + β, where γ and β are learned parameters which then adjust the hidden outputs to any mean and standard deviation.
Do γ and β “undo” the eﬀects of batch normalization? (why?)