I saw the wording of "batch normalization":

"Ideally, the normalization would be conducted over the entire training set, but to use this step jointly with stochastic optimization methods, it is impractical to use the global information. Thus, normalization is restrained to each mini-batch in the training process."

I don't understand why. Why using the mean and variance of entire training set is impractical in batch normalization? Since we only need to calculate all the net activations $z_n^{(l)}$ of each layers. Does it cost a lot of time?


1 Answer 1


Yeah, you’re right. The high time cost is the issue.

The problem is that stochastic optimization updates weights after each batch, but this requires computing information about the entire training data. That global information changes after each update of the parameters—that is, after each batch is processed.

For each batch (which can be as small as one training example!), you would have to scan through the entire training set to compute the normalization weights. That means that one training epoch is no longer a linear number of operations, with respect to the size of the training set. Instead, it becomes quadratic! For every epoch, for every batch in the training set, loop over the entire training set to compute these statistics.

Remember that these depend on the parameter values. You’re normalizing over values at the hidden layers, not simply computing model-agnostic functions of the data.

  • $\begingroup$ what is normalization weights here? Input data or parameters? $\endgroup$ Mar 25, 2021 at 2:11
  • $\begingroup$ Ah. When I say that, I’m referring to the empirical mean and variance of the activations at a given layer. In that way, it’s neither input data (because it depends on parameters) nor parameters (which usually refers to the things we learn with SGD). It’s just two statistics of the activations. (In a way, yes, these are parameters. But they’re special, and they’re treated differently.) $\endgroup$ Mar 25, 2021 at 12:26
  • $\begingroup$ sorry, suddenly confused, let me confirm one thing. batch normalization will normalize $\{z_1^{(l)},\cdots,z_N^{(l)}\}$ for all layers in each round of update during the whole training process, rather than only normalize at begin, right? And after training when we test an input $x_t,$ each layer, it should be normalized by the mean and variance of $\{z_1^{(l)^*},\cdots,z_N^{(l)^*}\}.$ Here $z_n^{(l)^*}$ is the net activation of samples under the trained parameter. Am I right? $\endgroup$ Mar 25, 2021 at 12:44
  • $\begingroup$ Your first sentence makes sense: any time the parameters change, the statistics about $z^{(\ell)}$ used for normalization will change. You never asked about testing, though. At test time, you don’t use statistics based on the test data. $\endgroup$ Mar 25, 2021 at 13:00
  • $\begingroup$ Thanks. Let me also confirm the process of test. Suppose $\{z_1^{(l)*},\cdots,z_N^{(l)*}\}$ is the net activation of entire samples under trained parameters with mean $\mu^{(l)*},$ variance $(\sigma^{(l)*})^2.$ Then for a test input $x,$ we have: $$x\rightarrow \hat{x}=\dfrac{x-\mu^{(0)*}}{\sigma^{(0)*}}\Rightarrow x^{(1)} \rightarrow \hat{x}^{(1)}=\dfrac{x^{(1)}-\mu^{(1)*}}{\sigma^{(1)*}}\Rightarrow x^{(2)} \rightarrow \hat{x}^{(2)}=\dfrac{x^{(2)}-\mu^{(2)*}}{\sigma^{(2)*}}\Rightarrow \cdots$$ Here $\Rightarrow$ is action by neutral network under the trained parameters. $\endgroup$ Mar 25, 2021 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.