Why does Batch Normalization need moving averages besides to track model accuracy?

Question

I was reading the new layer normalization (LN) paper and it mentioned that batch normalization (BN) batch normalization required moving averages. I was re-reading the paper but and it says the moving averages are needed to estimate the accuracy of the model as it trains (by accuracy I take it to mean train, validation, test errors/accuracy). I take the reason for this is so to save time since it probably estimates accuracies of mini batches and just averages it as it goes until the end.

However, the LA paper seems to imply (in the introduction) that activation accuracies are used in BN for some other reason. Why would BN require to store activation statistics at the end of training?

Answer 1

During evaluation, you may just have a single example, i.e. there is no batch to normalize over. In this case, you can use the average activations from training to do the normalization. The moving averages are because we may want to do some evaluation on held-out data while we are still training in order to detect convergence/over-fitting.