It is already common to do something "like"**(see asterisks below) z-standardization of the outputs of one neural network layer before passing it to the next. z-standardization would transform the columns of $H_{\ell}W_{\ell} + \beta_{\ell}$ (where $\ell$ denotes a layer and $H$ denotes a "hidden" matrix containing the values of hidden neurons, or the input data) to have 0 mean and unit standard deviation.

**In reality, batchnorm is used, which incorporates learnable weights to the standardization function to help the model "undo" or "modify" the deterministic nature of z-scaling.

(1) I observe that z-scoring is a nonlinear function in $W_{\ell}$, because we must compute the sample standard deviation of $H_{\ell}W_{\ell} + \beta_{\ell}$, which involves a square root. It follows that batch-norm will be nonlinear in the previous layer's weights as well.

(2) Thus, if we do batch norm, we do not need to use a common activation function such as ReLU or tanh to prevent the whole "stack" (composition) of affine layers from collapsing into one affine layer. Meanwhile, the community generally believes batch norm is "good". Thus, why not just use batch norm between layers and free ourselves from choosing between ReLU, ELU, etc. etc.?

^^ that is the purpose of my question

what follows is an observation of why my question might also be useful:

(3) Then, we may observe all the questions online about whether batch norm should be used before or after the activation function. But couldn't we just use batch norm and avoid this question?

Insights and/or references on this specific topic would be much appreciated!


  • $\begingroup$ I'm not sure what information you're seeking. My best guess is that your question is premised on a misunderstanding of how batch norm works. The point of batch norm is to use running mean and running standard deviation estimates; these estimates are used to compensate for the shifting means and standard deviations of inputs to the norm layer which occur because the network is training. Does this answer your question, or do you need clarification about a different component of batch norm? (What component do you wish to understand in more detail?) $\endgroup$ – Sycorax Sep 16 at 20:04
  • $\begingroup$ @Sycorax. In some sense, that's the point of my question. I understand the original motivation behind batch norm, but couldn't it also double as a nonlinearity? I mean, why do we need to compose something like a sigmoid, a relu, an elu etc. etc. with a z-standardization or its glorified cousin batch norm? $\endgroup$ – RMurphy Sep 17 at 13:59
  • $\begingroup$ A $z$ score is just a linear transformation of the inputs; if this is unclear, note that you can re-write $\frac{x - \mu}{\sigma}$ as $\frac{1}{\sigma}x - \frac{\mu}{\sigma}$. The duplicate question addresses why neural networks use nonlinear activation functions instead of linear functions: linear functions are closed under composition, so a network of linear functions is simply a linear model. $\endgroup$ – Sycorax Sep 17 at 14:42
  • $\begingroup$ @Sycorax, please, it is not a duplicate question. I understand perfectly well why nonlinear functions are not used, in terms of the answer you have given me above. Also, I do not agree that computing a standard deviation is linear in its arguments. The x you have written will have weights from the previous layer, and we will evaluate a square root of those weights. $\endgroup$ – RMurphy Sep 17 at 19:24
  • 1
    $\begingroup$ My advice is to use the edit button rewrite your question to clearly articulate what you know, what you would like to know, and where you are stuck. Right now, I can't make heads or tails of what you're trying to ask and what you would like to know. $\endgroup$ – Sycorax Sep 17 at 19:33

This two-layer network is linear for some input $z$: $$ \hat{y} = W_2(W_1z+b_1)+b_2. $$ We can rewrite it as $$ \begin{align} \hat{y}&=\underbrace{W_2W_1}_{W^\prime}z+\underbrace{W_2b_1 +b_2}_{b^\prime} \\ &= W^\prime z+b^\prime \end{align} $$ where all weights $W$ and biases $b$ are shaped appropriately and $z\in\mathbb{R}^d$. This works because linear functions are closed under composition.

Suppose we have $N$ observations of our data $x_1, x_2, \dots x_N$. If we apply operations to enforce a 0 mean and variance 1 at each step, then hidden layer $1$ is computed as

$$ \begin{align} h_1(z) &= (W_1 z + b_1 - \mu_1)(\sigma_1^{-1}I) \\ &=W_1(\sigma_1^{-1}I)z+(b_1 - \mu)(\sigma_1^{-1}I)\\ \mu_{j1} &= \frac{1}{N} \sum_i^N (W_1 x_{ji}+b_1) \\ \sigma_{j1}^2 &= \frac{1}{N} \sum_i^N ((W_1 x_{ji}+b_1) - \mu_1)^2 \\ \end{align} $$

This is linear in $z$, because $\sigma_1$ doesn't involve $z$. (The parameters $W_1, b_1, \mu_1$ also do not involve $z$, and operations with them are all linear. Only $\sigma_1$ requires a nonlinear operation, namely a square root, but it is the square root of $x$, not $z$.) We can do the same thing for the second layer $h_2$, and we will reach the same conclusion as we did in the two-layer case with no normalization applied. This observation answers all of your explicit questions.

The only difference between this scheme and batch norm is that batch norm updates the normalization for each batch and has additional location and scale parameters which are updated by gradient descent. However, these details obscure more than they reveal.

  • Estimating the mean and variance per batch is just an incremental step, and does not change the fact that $z$ scores are linear.
  • Likewise, the additional rescaling or shifting from batch norm is linear.

Implicitly, your question seems to be "Why do we want to have the outputs of nonlinear activation functions be contained around 0?" The answer is that activation functions are sensitive to the location and scale of the inputs.

  • Consider the function $\tanh(x)$. It has saturating linearities far from 0. For inputs sufficiently far from zero, the outputs will be essentially the same; if $x\ll0$, the outputs will all be close to $-1$, and if $x \gg 0$ the output will all be close to $1$. This is why keeping the mean close to 0 is important. Keeping the variance close to 1 is important because it will tend to keep $\tanh(x)$ in the range where $\tanh(x)$ is approximately linear, i.e. roughly in the interval $[-2,2]$.
  • Consider the function $\text{elu}(x)$. Suppose the inputs to the function are distributed as $\mathcal{N}(-100,1^2)$. The output will likewise have very little variation, because of the horizontal asymptote to the left. We would like to shift the input to the domain where the function is not nearly constant, i.e. around 0. Likewise, the scale of the inputs is important because it controls how positive the outputs can be (because this function is unbounded on the right).
  • The auxiliary weights and biases in batch norm compromise on both of these; the reasoning is that a neural network will lose its nonlinearity if all of the inputs are constrained to the linear portion of the activation function. Allowing these parameters to "override" or "counter" the effect of maintaining mean 0 and variance 1 is a compromise that improves model fit, because arbitrary nonlinear functions can be accommodated. The model only collapses to a linear model if that is the best fit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.