# Batch normalization variance calculation

In batch normalization the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$)

$\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during the test time we calculate the population statistics for the same as

$E[x] = E_B[\mu_B]$ and $Var[x] = \frac{m}{m-1}E_B[\sigma_B^2]$ (as you can see the unbiased estimate of population variance is calculated during the test time as we are building the model to predict the population distribution)

Would it not seem more logical that bias correction be done during training phase itself and subsequently its expected value can be used without the $\frac {m}{m-1}$ correction factor. i.e

During training:

$\sigma_B^2 = \frac 1{m-1} \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during test time:

$E[x] = E_B[\mu_B]$ and $Var[x] = E_B[\sigma_B^2]$

• Hi! I find the same issue very confusing and I'm just discovering now how different NN frameworks are vague /inconsistent about this point. 5years later, did you arrive at any conclusion??
– a06e
Commented Dec 6, 2022 at 17:23