I am trying to understand how batch normalization works. If I understand it correctly, when using batch normalization in a certain layer, the activations of all units/neurons in that layer are normalized to having zero mean and unit variance. Is that correct?

If so, how do I calculate the mean and covariance? As a main source I used the deep learning book by Bengio et al. It says the following:

Let $H$ be a minibatch of activations of the layer to normalize, arranged as a design matrix, with the activations for each example appearing in a row of the matrix. To normalize $H$ , we replace it with

$H' = \frac{H - \mu}{\sigma}$

where $\mu$ is a vector containing the mean of each unit and σ is a vector containing the standard deviation of each unit. Within each row of the matrix $H$, the arithmetic is element-wise, so $H_{i,j}$ is normalized by subtracting $\mu_j$ and dividing by $\sigma_j$.

For me this sounds as if they normalize the activation of a unit by subtracting the mean of that unit calculated over all examples in the minibatch and dividing by the standard deviation of that unit calculated over all examples in the minibatch.

However, in the following the book says:

At training time, $\mu = \frac{1}{m} \sum_i H_{i,:}$

If we normalize a unit using the mean of that unit calculated over all examples in the minibatch, shouldn't the formula be $\mu = \frac{1}{m} \sum_i H_{:,j}$?


1 Answer 1


I have solved the question myself. We calculate the mean and variance for each unit individually, using the entire mini-batch. We then use the mean and variance to normalize the activation of the unit. Also the formula is correct. Even if it looks different at first glance, it will give us a vector containing the mean for every unit.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.