# Batch normalization - how to compute mean and standard deviation

I am trying to understand how batch normalization works. If I understand it correctly, when using batch normalization in a certain layer, the activations of all units/neurons in that layer are normalized to having zero mean and unit variance. Is that correct?

If so, how do I calculate the mean and covariance? As a main source I used the deep learning book by Bengio et al. It says the following:

Let $$H$$ be a minibatch of activations of the layer to normalize, arranged as a design matrix, with the activations for each example appearing in a row of the matrix. To normalize $$H$$ , we replace it with

$$H' = \frac{H - \mu}{\sigma}$$

where $$\mu$$ is a vector containing the mean of each unit and σ is a vector containing the standard deviation of each unit. Within each row of the matrix $$H$$, the arithmetic is element-wise, so $$H_{i,j}$$ is normalized by subtracting $$\mu_j$$ and dividing by $$\sigma_j$$.

For me this sounds as if they normalize the activation of a unit by subtracting the mean of that unit calculated over all examples in the minibatch and dividing by the standard deviation of that unit calculated over all examples in the minibatch.

However, in the following the book says:

At training time, $$\mu = \frac{1}{m} \sum_i H_{i,:}$$

If we normalize a unit using the mean of that unit calculated over all examples in the minibatch, shouldn't the formula be $$\mu = \frac{1}{m} \sum_i H_{:,j}$$?