Reason for new question: I have a question about batch normalization as well as nonlinear "activation functions" in a neural network.  However, web searching this leads to a deluge of whether batch norm should be before or after an activation which is not my concern.

Despite being marked as a duplicate, this is not a duplicate, please see my comment below.

Moreover, we can simplify the discussion to just z-standardization; that is, centering to mean 0 and scaling to unit standard deviation.

z-scaling should be nonlinear in the weights of the layers that come before it, due to square and square root operations.

A motivation for using nonlinearities is that they prevent the whole composition of affine layers from collapsing into one affine layer.  Isn't standardization enough to avoid this? 

The discussion generalizes to batch-norm, where learnable scalings and centerings are employed.

Insights and/or references on this specific topic would be much appreciated!