4
$\begingroup$

I read in {1} section "5.2.5 SATURATION AND DEAD NEURONS" page 61:

The batch-normalization techniques became a key component for effective training of deep networks in computer vision. As of this writing, it is less popular in natural language applications.

Why are batch-normalization techniques less popular in natural language applications than in computer vision?


References:

  • {1} Goldberg, Yoav. "Neural network methods for natural language processing." Synthesis Lectures on Human Language Technologies 10, no. 1 (2017): 1-309.
$\endgroup$
1
$\begingroup$

I think the main reason is that computer vision models tend to be much deeper than the ones commonly used in NLP. It's rare to have more than three or four layers for NLP tasks and oftentimes you can get by with just a single layer LSTM. Batch normalization helps train deeper networks but it is not as important for shallower ones.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.