In the original Batch Norm paper (Ioffe and Szegedy 2015), the autors define Internal Covariate Shift as the "the change in the distributions of internal nodes of a deep network, in the course of training". They then present Batch Norm as a solution to address this issue by "normalizing layer inputs" across each mini-batch.

From my understanding, this "internal covariate shift" is the exact same issue that is typically addressed when designing our weight initializaiton criteria. For instance, in Kaiming initialization (He et al. 2015), "the central idea is to investigate the variance of the responses in each layer", so to "avoid reducing or magnifying the magnitudes of input signals exponentially". As far as I can tell, this is also addressing internal covariate shift.

Is my understanding correct? If this is the case, why do we often make use of both techniques? It seems redundant. Perhaps two solutions is better than one? If my understanding is incorrect, please let me know.

Thank you in advance.


Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." International conference on machine learning. PMLR, 2015.

He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." Proceedings of the IEEE international conference on computer vision. 2015.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.