I read in {1} section "5.2.5 SATURATION AND DEAD NEURONS" page 61:

The batch-normalization techniques became a key component for effective training of deep networks in computer vision. As of this writing, it is less popular in natural language applications.

Why are batch-normalization techniques less popular in natural language applications than in computer vision?


  • {1} Goldberg, Yoav. "Neural network methods for natural language processing." Synthesis Lectures on Human Language Technologies 10, no. 1 (2017): 1-309.

I think the main reason is that computer vision models tend to be much deeper than the ones commonly used in NLP. It's rare to have more than three or four layers for NLP tasks and oftentimes you can get by with just a single layer LSTM. Batch normalization helps train deeper networks but it is not as important for shallower ones.


I've been wondering about this as well. For some reason, applying batchnorm degrades the performance (accuracy) of NLP benchmarks most of the time. There is a recent paper trying to attribute this to the variance of weights that we are training.

We find that there are clear differences in the batch statistics of NLP data versus CV data. In particular, we observe that batch statistics for NLP data have a very large variance throughout training. This variance exists in the corresponding gradients as well. In contrast, CV data exhibits orders of magnitude smaller variance. See Figure 2 and 3 for a comparison of BN in CV and NLP.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.