2
$\begingroup$

I'm trying to make sense of this batch normalization (1) paper, in Section 3.2, it says

We could have also normalized the layer inputs u, but since u is likely the output of another nonlinearity, the shape of its distribution is likely to change during training, and constraining its first and second moments would not eliminate the covariate shift. In contrast, Wu + b is more likely to have a symmetric, non-sparse distribution, that is “more Gaussian” (Hyvarinen & Oja, 2000); normalizing it is likely to produce activations with a stable distribution.

Why the output of a non-linearity $u$ is likely to change during training (as opposed to $Wu+b$)?

Because the input of the non-linearity is in the form of $Wu+b$ as well, if the input is stable or “more Gaussian”, why is the output likely to change?

1: Ioffe S. and Szegedy C. (2015), "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015. Journal of Machine Learning Research: W&CP volume 37

Improve this question
$\endgroup$
3
  • $\begingroup$ Did you find anything useful that address any aspect of your question? Thanks for asking such a good question, its been a very mysterious remark from the paper for me too. $\endgroup$
    – Charlie Parker
    Jun 13, 2016 at 0:47
  • $\begingroup$ For me what confuses me is that I thought that covariate shift would be (and should be) present in both $Wu+b$ and $u$, since for both of them, parameter values change during training, so the distributions for both should change. So my gut instinct is to BN both (i.e. BN(Wu) and BN(u) but I guess thats not the preferred thing to do for some reason). $\endgroup$
    – Charlie Parker
    Jun 13, 2016 at 0:53
  • $\begingroup$ @CharlieParker thanks for replying, I still haven't figured it out. for your question, I guess the author probably means that, covariate shift would be present in both terms as you said, but BN(u) is not helpful with eliminating the covariate shift, so only BN(wu) is used (because wu is more Gaussian). $\endgroup$
    – dontloo
    Jun 13, 2016 at 2:48

1 Answer 1

Reset to default
3
$\begingroup$

The shape of the distribution of $Wu+b$ being not likely to change is because of the central limit theorem, which states that,

given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined (finite) expected value and finite variance, will be approximately normally distributed, regardless of the underlying distribution.

There's a nice illustration on the wikipedia page Illustration of the central limit theorem,

  1. Original probability density function           2. Density of a sum of two variables

  1. Density of a sum of three variables            4. Density of a sum of four variables

As the shape of distribution of $Wu+b$ is "more Gaussian", after applying batch normalization it will become close to $N(0,1)$, regardless what the distribution of $u$ is.

So by "mapping" some internal distributions to $N(0,1)$ batch normalization helps addressing the internal covariate shift problem.

On the other hand if we apply batch normalization to $u$ or $\sigma(Wu+b)$ instead, as the shape of the distribution is likely to change, normalizing the first and second moments will not result in a stable distribution.

Improve this answer
$\endgroup$
4
  • $\begingroup$ it would be nice to actually plot the distribution of the pre-activations after they are processed by BN during training to see if this actually holds. For example, W sort of needs to be a constant for this to hold but its changing randomly according to the expectation of the gradient. I wonder how much that changes this very plausible argument (hypothesis). $\endgroup$
    – Charlie Parker
    Aug 13, 2016 at 15:43
  • $\begingroup$ @CharlieParker yea, that would be nice $\endgroup$
    – dontloo
    Aug 13, 2016 at 16:27
  • 2
    $\begingroup$ BTW, what's the meaning of non-sparse distribution? $\endgroup$
    – n0p
    Sep 20, 2016 at 18:00
  • $\begingroup$ @n0p I guess a sparse distribution is a distribution whose pdf is zero almost everywhere (e.g. the distribution in figure 1), whereas Gaussian's pdf (like figure 4) is spread over the whole space (though the value can be rather small), so it's not sparse. $\endgroup$
    – dontloo
    Sep 21, 2016 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.