How does Batch Normalization not lead to the model blowing up? [duplicate]

Question

I was reading the Batch Norm paper and in this paragraph,

We could consider whitening activations at every training step or at some interval, either by modifying the network directly or by changing the parameters of the optimization algorithm to depend on the network activation values (Wiesler et al., 2014; Raiko et al., 2012; Povey et al., 2014; Desjardins & Kavukcuoglu). However, if these modifications are interspersed with the optimization steps, then the gradient descent step may attempt to update the parameters in a way that requires the normalization to be updated, which reduces the effect of the gradient step. For example, consider a layer with the input $u$ that adds the learned bias $b$, and normalizes the result by subtracting the mean of the activation computed over the training data: $\hat{x} = x − E[x]$ where $x = u + b$, $\mathcal{X} = \{x_{1 \ldots N} \}$ is the set of values of $x$ over he training set, and $E[x] = \frac{1}{N} \sum_{i=1}^N x_i$ . If a gradient descent step ignores the dependence of $E[x]$ on $b$, then it will update $b \leftarrow b + \Delta b$, where $\Delta b \propto −\partial l/ \partial \hat{x}$. Then $u + (b + \Delta b) − E[u + (b + \Delta b)] = u + b − E[u + b]$. Thus, the combination of the update to $b$ and subsequent change in normalization led to no change in the output of the layer nor, consequently, the loss. As the training continues, $b$ will grow indefinitely while the loss remains fixed. This problem can get worse if the normalization not only centers but also scales the activations. We have observed this empirically in initial experiments, where the model blows up when the normalization parameters are computed outside the gradient descent step.

What's the point of having $b$ anyway? (Later in the paper, it says that b is removed during batch normalization, so what's the aim of the math shown in this picture and how is this* math different from the one that we do during actual batch normalization).

This happens later in the paper,

Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by β in Alg. 1). Thus, z = g(Wu + b) is replaced with z = g(BN(Wu))

Answer 1

The explanation comes in the next paragraph. Instead of normalizing by the mean of activations in a single batch, they normalize with respect to the whole training data ($\mathcal{X}$). The way they achieve this is by using moving averages that get updated for each mini-batch during training.

The reason why the bias term becomes redundant is due to learned scale and shift variables $\gamma^{(k)}$, and $\beta^{(k)}$. Without these, the normalization would squash everything which could limit what a non-linearity activation function can represent. They give the example of the sigmoid activation function which would be constraint to the linear part only and the tails would be squashed.