What do they mean by "batchnormalization allows to initialization of weights less carefully?"

Question

In Towards Data Science - Manish Chablani - Batch Normalization, it is stated that:

Makes weights easier to initialize — Weight initialization can be difficult, and it’s even more difficult when creating deeper networks. Batch normalization seems to allow us to be much less careful about choosing our initial starting weights.

In the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, it is stated that:

Batch Normalization allows us to use much higher learning rates and be less careful about initialization.

And:

In practice, the saturation problem and the resulting vanishing gradients are usually addressed by using Rectified Linear Units (Nair & Hinton, 2010) ReLU(x) = max(x, 0), careful initialization (Bengio & Glorot, 2010; Saxe et al., 2013), and small learning rates. If, however, we could ensure that the distribution of nonlinearity inputs remains more stable as the network trains, then the optimizer would be less likely to get stuck in the saturated regime, and the training would accelerate.

Suppose that we are on the first layer, $x$ is our input data. so the output ($y$) of that layer is $y=activation(BN(Wx)))$ in forward pass. My question is about: If we initialize our weight to higher values, does BN helps us with exploding gradients issue? Suppose that we use sigmoid activation.

I understand that in the forward pass since we apply BN after $Wx$, it will reduce the result that we obtained in $BN(Wx)$. However, in the backward pass, while calculating the derivate of loss w.r.t to the first layer's weights ($W1$), according to the chain rule, all the weights of the following layers $(W2, W3, ..)$ will be included in the formula (dot product). Therefore we will suffer from large initialization I think because these high gradient values will be used to update the actual $W1$.

How BN makes initializations easier based on the paper and the link that I shared? What do they want to say precisely?

Answer 1

In A Gentle Introduction to Batch Normalization for Deep Neural Networks, Jason Brownlee suggests "The stability to training brought by batch normalization can make training deep networks less sensitive to the choice of weight initialization method". In other words, batch normalisation acts as a form of regularisation, reducing model variance - including the variance due to different weight initialisations.

Another way of describing this is in terms of the loss surface. As neural networks are complex, they can have a complex loss surface, with lots of local minima. So a slight change in the initial weights values can lead to a different local minimum (in other words, a different solution). Santurkar et al. How Does Batch Normalization Help Optimization? found that batch normalisation leads to a smoother loss surface, so there are fewer local minima. Therefore models with different weight initialisations are more likely to converge to the same solution when trained.