I was reading the batch normalization (BN) paper (1) and didn't understand the need to use moving averages to track the accuracy of the model and even if I accepted that it was the right thing to do, I don't understand what they are doing exactly.

To my understanding (which my be wrong), the paper mentions that it uses the population statistics rather than the mini-batch, statistics once the model has finished training. After some discussion of unbiased estimates (that seems tangential to me and don't understand why it talks about that) they go and say:

Using moving averages instead, we track the accuracy of the model as it trains.

That is the part that is confusing to me. Why do they do moving averages to estimate the accuracy of the model and over what data set?

Usually what people do to estimate the generalization of their model, they just track the validation error of their model (and potentially early stop their gradient descent to regularize). However, it seems that batch normalization is doing something completely different. Can someone clarify what and why its doing something different?

1: Ioffe S. and Szegedy C. (2015),
"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift",
Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.
Journal of Machine Learning Research: W&CP volume 37


When using batch_normalization first thing we have to understand is it work on two different ways when in Training and Testing .

  1. In Training we need to calculate mini batch mean in order to normalize the batch

  2. In the inference we just apply pre-calculated mini batch statistics

So in the 2nd thing how to calculate this mini batch statics

Here comes the moving average

running_mean = momentum * running_mean + (1 - momentum) * sample_mean
running_var = momentum * running_var + (1 - momentum) * sample_var

I think it's talking about using moving average to estimate the training accuracy as it trains, which can be used for other mini-batch based training as well, not necessarily for BN.

For neural networks knowing the training error can help determining say when to stop or slow down the learning rate.

Since they use mini-batch training, it would be inefficient to compute the training accuracy over the entire training set after every iteration, using the moving average over the mini-batches instead can be a good approximation.

In fact for BN there's another usage of the moving average, it is to estimate the population mean and variance as it trains. But this does not affect the training process, the population mean and variance are only used at test time. This is just a trick that saves us from computing the population mean and variance layer by layer.

  • $\begingroup$ so how do you check the test accuracy as it trains? Also with a moving average? $\endgroup$ – Pinocchio Jun 21 '16 at 21:28
  • $\begingroup$ @Pinocchio mostly I just check the test accuracy once every some thousand of iterations. $\endgroup$ – dontloo Jun 22 '16 at 1:41
  • $\begingroup$ but when you check it, do you check it on the whole test set? I'm just confused when moving averages are suppose to be used in BN, is it for checking (train,cv,test?) accuracies? To get population means? To average parameters, just when are moving averages used and why? $\endgroup$ – Pinocchio Jun 22 '16 at 4:25
  • $\begingroup$ @Pinocchio yes the whole test set. and I updated the answer a bit and added another possible usage of moving average for BN, i dont think it's mentioned in the paper but it's commonly used. $\endgroup$ – dontloo Jun 22 '16 at 6:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.