In the training phase of Batch normalization we keep moving average of the activations in every layer so that we can use those "moving averages" in the test time. My question is why do we need to take moving average? We can approximate the mean by taking the mean of the activations in every layer in each iteration and then dividing by the number of batches and then use it in the test time. For example, why not do something like:
AVG = 0
while(# of iterations of training)
AVG = AVG + mean(activations of 3rd hidden layer)
AVG_used_in_test = AVG / number of iterations
instead of the moving average?