In this video and this answer, it's mentioned that batch normalization doesn't allow the mean and variance of the parameters of any particular hidden layer to vary too much with change in previous layer's node values. Let's say we have a network like this (taken from the video):

enter image description here

As we change the mini-batch that the neural network is trained on, it's possible that the distribution of the input features with respect to different mini-batches is different. As far as I understand, this is called covariate shift. An issue with covariate shift is that maybe the distribution of input features with respect to the validation set examples is different from the training set (ref. black cats / colored cats example in the linked video), and the distributions w.r.t. the different mini-batches are different.

Consequently, since the learned weights connecting the input layer to the next layer depend on the input features distribution, the learned weights will accordingly change with each mini-batch. Thus the activations in the next hidden layer will also vary in their distribution w.r.t. different mini-batches.

First question: Is my understanding so far correct?

Now I can understand that limiting the mean and variance of input features or that of the hidden layer activations will address the covariate shift problem only if the distributions across mini-batches (or between training and validation sets) are fundamentally the same but differ in parameters (not the neural network parameters - I mean mean and stddev). Crudely speaking, if the distribution of input features w.r.t different mini-batches is normal, but the parameters (mean and stddev) are different, then batch normalization seems fine.

Question 2: But what if the input features have normal distribution w.r.t. one mini-batch and a completely different distribution w.r.t. another? I don't see how limiting the mean and variance will help. Maybe I'm ignoring some key assumptions.

Question 3: Why does limiting the variation in the distributions of the hidden layers speed up the convergence?

  • $\begingroup$ I'm not certain that batch norm is completely understood at this time. The bibliography at stats.stackexchange.com/questions/352036/… suggests that there is some controversy over how and why batch norm improves models and model estimation. $\endgroup$ – Reinstate Monica Jul 23 at 18:45
  • $\begingroup$ @Sycorax: So the efficacy of batch norm is only understood empirically, then? $\endgroup$ – Shirish Kulhari Jul 23 at 18:49
  • $\begingroup$ I don't know for sure. I'm uncertain that the matter is entirely settled. It's worth asking the question in case someone who knows more about it than I do arrives and answers. $\endgroup$ – Reinstate Monica Jul 23 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.