I'm trying to make sense of this batch normalization (1) paper, in Section 3.2, it says
We could have also normalized the layer inputs u, but since u is likely the output of another nonlinearity, the shape of its distribution is likely to change during training, and constraining its first and second moments would not eliminate the covariate shift. In contrast, Wu + b is more likely to have a symmetric, non-sparse distribution, that is “more Gaussian” (Hyvarinen & Oja, 2000); normalizing it is likely to produce activations with a stable distribution.
Why the output of a non-linearity $u$ is likely to change during training (as opposed to $Wu+b$)?
Because the input of the non-linearity is in the form of $Wu+b$ as well, if the input is stable or “more Gaussian”, why is the output likely to change?
1: Ioffe S. and Szegedy C. (2015), "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015. Journal of Machine Learning Research: W&CP volume 37