Reason for new question: I have a question about batch normalization as well as nonlinear "activation functions" in aIt is already common to do something "like"**(see asterisks below) z-standardization of the outputs of one neural network. However, web searching this leads layer before passing it to the next. z-standardization would transform the columns of $H_{\ell}W_{\ell} + \beta_{\ell}$ (where $\ell$ denotes a delugelayer and $H$ denotes a "hidden" matrix containing the values of whether batch norm should be beforehidden neurons, or after an activation which is not my concernthe input data) to have 0 mean and unit standard deviation.
Despite being marked as a duplicate**In reality, thisbatchnorm is not a duplicate, please see my comment below.
Moreoverused, we can simplifywhich incorporates learnable weights to the discussionstandardization function to justhelp the model "undo" or "modify" the deterministic nature of z-standardization; that is, centering to mean 0 and scaling to unit standard deviationscaling.
(1) I observe that z-scaling should bescoring is a nonlinear function in $W_{\ell}$, because we must compute the weightssample standard deviation of the layers that come before it$H_{\ell}W_{\ell} + \beta_{\ell}$, due to square andwhich involves a square root operations. It follows that batch-norm will be nonlinear in the previous layer's weights as well.
A motivation for using nonlinearities is that they(2) Thus, if we do batch norm, we do not need to use a common activation function such as ReLU or tanh to prevent the whole composition"stack" (composition) of affine layers from collapsing into one affine layer. Isn't standardization enough to avoid this Meanwhile, the community generally believes batch norm is "good". Thus, why not just use batch norm between layers and free ourselves from choosing between ReLU, ELU, etc. etc.?
The discussion generalizes to batch-norm^^ that is the purpose of my question
what follows is an observation of why my question might also be useful:
(3) Then, where learnable scalings and centerings are employedwe may observe all the questions online about whether batch norm should be used before or after the activation function. But couldn't we just use batch norm and avoid this question?
Insights and/or references on this specific topic would be much appreciated!
Thanks