I was reading the new layer normalization (LN) paper and it mentioned that batch normalization (BN) batch normalization required moving averages. I was re-reading the paper but and it says the moving averages are needed to estimate the accuracy of the model as it trains (by accuracy I take it to mean train, validation, test errors/accuracy). I take the reason for this is so to save time since it probably estimates accuracies of mini batches and just averages it as it goes until the end.

However, the LA paper seems to imply (in the introduction) that activation accuracies are used in BN for some other reason. Why would BN require to store activation statistics at the end of training?

  • $\begingroup$ does this answer help (the update part)? $\endgroup$ – dontloo Aug 15 '16 at 1:46
  • $\begingroup$ @dontloo oh, I see that you updated that answer. I guess I missed that. However, I think I need some more details to understand that point, I still feel I don't appreciate why thats important. $\endgroup$ – Pinocchio Aug 15 '16 at 9:55

During evaluation, you may just have a single example, i.e. there is no batch to normalize over. In this case, you can use the average activations from training to do the normalization. The moving averages are because we may want to do some evaluation on held-out data while we are still training in order to detect convergence/over-fitting.

  • $\begingroup$ but we can also "do some evaluation on held-out data while we are still training" by dividing by the number of iterations.for example: after 5 iterations we will divide by 5 and after 10 we'll divide by 10 ....etc. am i right? $\endgroup$ – floyd Sep 19 '17 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.