# Batch normalization - how to compute mean and standard deviation

I am trying to understand how batch normalization works. If I understand it correctly, when using batch normalization in a certain layer, the activations of all units/neurons in that layer are normalized to having zero mean and unit variance. Is that correct?

If so, how do I calculate the mean and covariance? As a main source I used the deep learning book by Bengio et al. It says the following:

Let $$H$$ be a minibatch of activations of the layer to normalize, arranged as a design matrix, with the activations for each example appearing in a row of the matrix. To normalize $$H$$ , we replace it with

$$H' = \frac{H - \mu}{\sigma}$$

where $$\mu$$ is a vector containing the mean of each unit and σ is a vector containing the standard deviation of each unit. Within each row of the matrix $$H$$, the arithmetic is element-wise, so $$H_{i,j}$$ is normalized by subtracting $$\mu_j$$ and dividing by $$\sigma_j$$.

For me this sounds as if they normalize the activation of a unit by subtracting the mean of that unit calculated over all examples in the minibatch and dividing by the standard deviation of that unit calculated over all examples in the minibatch.

However, in the following the book says:

At training time, $$\mu = \frac{1}{m} \sum_i H_{i,:}$$

If we normalize a unit using the mean of that unit calculated over all examples in the minibatch, shouldn't the formula be $$\mu = \frac{1}{m} \sum_i H_{:,j}$$?

It is worth saying that the BatchNorm normalization equations vary across literature, and I conjecture that the variations are not completely equivalent. The form presented in the DL book (here being discussed) can be contrasted with the one given in this book or Ng videos on youtube. Specifically, while several authors mentioned the sample mean, i.e., the mean of the entire batch, DL book presents BatchNorm using vectors.

Let see if we can unravel the DL book notation.

First, we have to consider the design matrix $$\boldsymbol{H}$$. What are the rows and the column of this matrix? Well, in my opinion, that is ambiguous in the DL book:

with the activations for each example appearing in a row of the matrix

Are the rows the activation $$a_{i,1}, a_{i,2}, \cdots, a_{i,n}$$ for one training example $$\boldsymbol{x}^{(i)}$$? In other words,

1. the activations at layer $$l$$ for one training example are arranged in a row $$i$$.

Or

are the rows the activation $$a_{i,1}, a_{i,2}, \cdots, a_{i,n}$$ for all training example $$\boldsymbol{x}^{(i)}, \boldsymbol{x}^{(i + 1)}, \cdots, \boldsymbol{x}^{(m)}$$? This means that

1. the activations at layer $$l$$ for one training example are arranged in a column $$j$$.

I will stitch to 2) for reasons that will become apparent next. Let see a concrete design matrix of three examples being trained in a layer with three units:

$$\begin{bmatrix} a_{1,1} & a_{1,2} & a_{1,2}\\ a_{2,1} & a_{2,2} & a_{2,3}\\ a_{3,1} & a_{3,2} & a_{33}\\ \end{bmatrix}$$

Second, since we need a mean vector $$\boldsymbol{\mu}$$ and std. dev. vector $$\boldsymbol{\sigma}$$, the notation cannot refer to the sample mean because that would be a scalar.

where $$\boldsymbol{\mu}$$ is a vector containing the mean of each unit

Now, the question is if we must average over rows or columns. Since we considered that one column represents the same unit, we have

\begin{align}\label{eq:average} \mu_1 &= \frac{1}{3} (a_{1,1} + a_{2,1} + a_{3,1})\\ &= \frac{1}{m} \sum_{i = 1}^m a_{i,1}\\ \mu_2 &= \frac{1}{m} \sum_{i = 1}^m a_{i,2}\\ \mu_3 &= \frac{1}{m} \sum_{i = 1}^m a_{i,3}.\\ \end{align}

Correspondinly,

\begin{align} \sigma_1 &= \sqrt \frac{|a_{1,1} - \mu_1|^2 + |a_{2,1} - \mu_1|^2 + |a_{3,1} - \mu_1|^2}{3}\\ &= \sqrt \frac{\sum^m_{i = 1} |a_{i,1} - \mu_1|^2}{m}\\ \sigma_2 &= \sqrt \frac{\sum^m_{i = 1} |a_{i,2} - \mu_2|^2}{m}\\ \sigma_3 &= \sqrt \frac{\sum^m_{i = 1} |a_{i,3} - \mu_3|^2}{m},\\ \end{align}

resulting in column vectors $$\boldsymbol{\mu} = \{\mu_1, \mu_2, \mu_3\}$$ and $$\boldsymbol{\sigma} = \{\sigma_1, \sigma_2, \sigma_3\}.$$

The arithmetic here is based on broadcasting the vector $$\boldsymbol{\mu}$$ and the vector $$\boldsymbol{\sigma}$$ to be applied to every row of the matrix $$\boldsymbol{H}$$.

I do not know what exactly that means, let us continue.

Within each row, the arithmetic is element-wise, so $$H_{i,j}$$ is normalized by subtracting $$\mu_j$$ and dividing by $$\sigma_j$$.

That would mean $$a_{1,1}$$ is normalized as $$\frac{a_{1,1} -\mu_1}{\sigma_1}, \cdots$$

$$\begin{bmatrix} \frac{a_{1,1} -\mu_1}{\sigma_1} & \frac{a_{1,2} -\mu_2}{\sigma_2} & \frac{a_{1,3} -\mu_3}{\sigma_3}\\ \frac{a_{2,1} -\mu_1}{\sigma_1} & \frac{a_{2,2} -\mu_2}{\sigma_2} & \frac{a_{2,3} -\mu_3}{\sigma_2}\\ \frac{a_{3,1} -\mu_1}{\sigma_1} & \frac{a_{3,2} -\mu_2}{\sigma_2} & \frac{a_{3,3} -\mu_3}{\sigma_3}, \end{bmatrix}$$

which indeed is well defined since the average was taken in per-column basis (see above).

It remains only the most difficult part:

At training time,

$$\boldsymbol{\mu} = \frac{1}{m} \sum_i \boldsymbol{H}_{i,:},$$

which I would interpret as

$$\boldsymbol{\mu} = \frac{1}{3} ( \boldsymbol{H}_{1,:} + \boldsymbol{H}_{2,:} + \boldsymbol{H}_{3,:}).$$

Here, according to the DL book notation, $$\boldsymbol{H}_{1,:}$$ is the first row, $$\boldsymbol{H}_{2,:}$$ the second row, etc. Let us remove the fration because it is not necessary at this point. This results in

\begin{align} \boldsymbol{\mu} &= ( \boldsymbol{H}_{1,:} + \boldsymbol{H}_{2,:} + \boldsymbol{H}_{3,:})\\ &= \{H_{1,1}, H_{1,2}, H_{1,3}\} + \{H_{2,1}, H_{2,2}, H_{2,3}\} + \{H_{3,1}, H_{3,2}, H_{3,3}\}\\ &= \{H_{1,1} + H_{2,1} + H_{3,1}, \; H_{1,2} + H_{2,2} + H_{3,2}, \;H_{3,1} + H_{3,2}, H_{3,3}\}\\ &= \frac{1}{3} \{a_{1,1} + a_{2,1} + a_{3,1}, \; a_{1,2} + a_{2,2} + a_{3,2}, \;a_{3,1} + a_{3,2}, a_{3,3}\}\\ \boldsymbol{\mu} &= \{\mu_1, \mu_2, \mu_3\},\\ \end{align}

where in the first three lines I removed the fraction. In the third line, I applied vector summation. Then, in the fourth line, I introduced again the fraction and substituted the matrix elements by the corresponding activations. Finally, in the fifth line, I substituted by the corresponding mean.

Which is, of course, also well-defined and corresponds to our assumption that activations of the same unit for a determined mini-batch example $$\boldsymbol{x}^{(i)}$$ are represented in one column of the design matrix $$\boldsymbol{H}$$.

It's crazy, I know.

The interpretation of the per-activations-example-column assumption is for me rather tricky. The intuition usually mentioned when presenting BatchNorm is that one would like to normalize hidden layers, similarly as to when one normalizes features at the input layer because it benefits learning. However, as I see it, normalization is being performed unit-wise and not feature-wise. I left open the question if these two are equivalent (I would say rather no). That is, is normalizing the output of the unit for all examples in the mini-batch equivalent to normalizing the output of all units (e.g. averaging over rows in $$\boldsymbol{H}$$) for one example in a mini-batch? Is this somehow a misconception? Finally, are these two equivalent to normalizing the entire dataset using the sample mean and std. dev.?

I have solved the question myself. We calculate the mean and variance for each unit individually, using the entire mini-batch. We then use the mean and variance to normalize the activation of the unit. Also the formula is correct. Even if it looks different at first glance, it will give us a vector containing the mean for every unit.