4
$\begingroup$

In the training phase of Batch normalization we keep moving average of the activations in every layer so that we can use those "moving averages" in the test time. My question is why do we need to take moving average? We can approximate the mean by taking the mean of the activations in every layer in each iteration and then dividing by the number of batches and then use it in the test time. For example, why not do something like: 

AVG = 0        
while(# of iterations of training)  
     AVG = AVG + mean(activations of 3rd hidden layer)        
AVG_used_in_test = AVG / number of iterations

instead of the moving average?

Improve this question
$\endgroup$
7
  • $\begingroup$ Can you provide any context for this? What are you asking about? $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 14, 2017 at 1:41
  • $\begingroup$ i changed the question details and added a reference to a paper. $\endgroup$
    – floyd
    Commented Sep 14, 2017 at 2:04
  • $\begingroup$ Thank you for doing so. I don't know this topic well enough to understand the question fully, but I suspect it will be clear enough to be answerable to people who work in that area. I also edited your question somewhat for more colloquial English. Please ensure it still says what you want it to. $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 14, 2017 at 2:42
  • $\begingroup$ What's wrong with using a moving average? $\endgroup$
    – Aaron
    Commented Sep 15, 2017 at 1:06
  • $\begingroup$ Aaron. i just want to know its intuition, why don't we just divide the activations by the number of iterations in the test time? $\endgroup$
    – floyd
    Commented Sep 19, 2017 at 21:33

1 Answer 1

Reset to default
1
$\begingroup$

Approximating the mean by taking the mean of the activations in every layer in each iteration and then dividing by the number of batches would give equal weight to all the batches. But all batches can't be given same weight because each batch won't be representing the training set because of sampling without replacement within each epoch. Also during training parameters get updated so in intial epochs the batch means wont be an accurate estimate because parameters will have random initialization and they might change rapidly in initial epochs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.