# What do they mean by "batchnormalization allows to initialization of weights less carefully?"

In Towards Data Science - Manish Chablani - Batch Normalization, it is stated that:

Makes weights easier to initialize — Weight initialization can be difficult, and it’s even more difficult when creating deeper networks. Batch normalization seems to allow us to be much less careful about choosing our initial starting weights.

In the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, it is stated that:

Batch Normalization allows us to use much higher learning rates and be less careful about initialization.

And:

In practice, the saturation problem and the resulting vanishing gradients are usually addressed by using Rectified Linear Units (Nair & Hinton, 2010) ReLU(x) = max(x, 0), careful initialization (Bengio & Glorot, 2010; Saxe et al., 2013), and small learning rates. If, however, we could ensure that the distribution of nonlinearity inputs remains more stable as the network trains, then the optimizer would be less likely to get stuck in the saturated regime, and the training would accelerate.

Suppose that we are on the first layer, $$x$$ is our input data. so the output ($$y$$) of that layer is $$y=activation(BN(Wx)))$$ in forward pass. My question is about: If we initialize our weight to higher values, does BN helps us with exploding gradients issue? Suppose that we use sigmoid activation.

I understand that in the forward pass since we apply BN after $$Wx$$, it will reduce the result that we obtained in $$BN(Wx)$$. However, in the backward pass, while calculating the derivate of loss w.r.t to the first layer's weights ($$W1$$), according to the chain rule, all the weights of the following layers $$(W2, W3, ..)$$ will be included in the formula (dot product). Therefore we will suffer from large initialization I think because these high gradient values will be used to update the actual $$W1$$.

How BN makes initializations easier based on the paper and the link that I shared? What do they want to say precisely?

• I think this is still an active area of research, but maybe this article could be of help: kyleluther.github.io/2020/02/18/… Mar 3, 2022 at 0:52