35
$\begingroup$

I was reading the batch normalization (BN) paper (1) and didn't understand the need to use moving averages to track the accuracy of the model and even if I accepted that it was the right thing to do, I don't understand what they are doing exactly.

To my understanding (which might be wrong), the paper mentions that it uses the population statistics rather than the mini-batch statistics once the model has finished training. After some discussion of unbiased estimates (that seems tangential to me and I don't understand why it talks about that) they go and say:

Using moving averages instead, we track the accuracy of the model as it trains.

That is the part that is confusing to me. Why do they do moving averages to estimate the accuracy of the model and over what data set?

Usually what people do to estimate the generalization of their model, they just track the validation error of their model (and potentially early stop their gradient descent to regularize). However, it seems that batch normalization is doing something completely different. Can someone clarify what and why it's doing something different?


1: Ioffe S. and Szegedy C. (2015),
"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift",
Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.
Journal of Machine Learning Research: W&CP volume 37

$\endgroup$
2
  • 2
    $\begingroup$ Are you satisfied with the upvoted answer? It isn't an "answer" at all, if you ask me; if still relevant, I can provide a better answer. $\endgroup$ Nov 8, 2019 at 20:26
  • 2
    $\begingroup$ @OverLordGoldDragon provide ur own answer :) $\endgroup$ Dec 27, 2019 at 5:05

3 Answers 3

13
$\begingroup$

When using batch_normalization first thing we have to understand is that it works on two different ways when in Training and Testing.

  1. In Training we need to calculate mini batch mean in order to normalize the batch

  2. In the inference we just apply pre-calculated mini batch statistics

So in the 2nd thing how to calculate this mini batch statics

Here comes the moving average

running_mean = momentum * running_mean + (1 - momentum) * sample_mean
running_var = momentum * running_var + (1 - momentum) * sample_var
$\endgroup$
3
  • 14
    $\begingroup$ This does not answer the question whatsoever; "why" and "how" was asked, and a superficial "what" was given. It shouldn't be upvoted. $\endgroup$ Nov 8, 2019 at 20:26
  • $\begingroup$ does the pre-calcualted stats come from data at training or test? $\endgroup$ Mar 6, 2020 at 8:53
  • $\begingroup$ @CharlieParker Training data, I would presume. The mini-batches are created during training and their mean/variance is also calculated during training, after all. $\endgroup$ Apr 9, 2020 at 0:06
6
$\begingroup$

They are talking about batch normalization, which they have described for the training procedure but not for inference.

This is a process of normalizing the hidden units using sample means etc.

In this section they explain what to do for the inference stage, when you are just making predictions ( ie after training has completed).

However, in stopped validation you interleave prediction on validation set with training to estimate your validation error.

So during this process you don't have a population average (the averages are still changing as you train), so then you use a running average to calculate the batch norm parameters to calculate performance on validation set.

It is in this sense that

Using moving averages instead, we track the accuracy of the model as it trains.

nothing to do with literally using the running means as a metric for neural network performance.

$\endgroup$
0
$\begingroup$

In the paper you referenced, the suggested test time behavior is to compute sample mean and variance for each feature using a large number of training images rather than using a running average.

This block of code

 running_mean = momentum * running_mean + (1 - momentum) * sample_mean
 running_var = momentum * running_var + (1 - momentum) * sample_var

represents an alternative approach for test time that doesn't require the extra estimation step needed in the paper. For the alternative moving average we just update the mean and variance using exponential decay model based on the momentum parameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.