Why are batch-normalization techniques less popular in natural language applications than in computer vision? I read in {1}  section "5.2.5 SATURATION AND DEAD NEURONS" page 61:

The batch-normalization techniques became
  a key component for effective training of deep networks in computer vision. As of this writing, it
  is less popular in natural language applications.

Why are  batch-normalization techniques less popular in natural language applications than in computer vision?

References:


*

*{1} Goldberg, Yoav. "Neural network methods for natural language processing." Synthesis Lectures on Human Language Technologies 10, no. 1 (2017): 1-309.

 A: I think the main reason is that computer vision models tend to be much deeper than the ones commonly used in NLP. It's rare to have more than three or four layers for NLP tasks and oftentimes you can get by with just a single layer LSTM. Batch normalization helps train deeper networks but it is not as important for shallower ones. 
A: I've been wondering about this as well. For some reason, applying batchnorm degrades the performance (accuracy) of NLP benchmarks most of the time. There is a recent paper trying to attribute this to the variance of weights that we are training.

We find that there are clear differences in the
batch statistics of NLP data versus CV data. In
particular, we observe that batch statistics for NLP
data have a very large variance throughout training.
This variance exists in the corresponding gradients
as well. In contrast, CV data exhibits orders of
magnitude smaller variance. See Figure 2 and 3 for
a comparison of BN in CV and NLP.

https://arxiv.org/pdf/2003.07845v1.pdf
