This paper says that the notion of a batch problematic for RNNs (page 9) (which is why you can't apply batch normalization for RNNs?). Why is it hard to talk about batches for RNNs?
Eg. the Pytorch API even has a parameter to control the batch size, so batches "clearly" make sense for RNNs (and, therefore, batch normalization also make sense).
What's going on?
The passage you refer to on page 9 says
Layer Normalization [1] performs normalization over the entire layer instead of the batch, which is suitable for contexts where the notion of a batch is problematic (e.g. recurrent neural networks).
This is a single-sentence summary of a whole article; the symbol [1] is a citation to Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. "Layer normalization." arXiv preprint arXiv:1607.06450, 2016.
The key part of the cited article says
The recent sequence to sequence models [Sutskever et al., 2014] utilize compact recurrent neural networks to solve sequential prediction problems in natural language processing. It is common among the NLP tasks to have different sentence lengths for different training cases. This is easy to deal with in an RNN because the same weights are used at every time-step. But when we apply batch normalization to an RNN in the obvious way, we need to to compute and store separate statistics for each time step in a sequence. This is problematic if a test sequence is longer than any of the training sequences. Layer normalization does not have such problem because its normalization terms depend only on the summed inputs to a layer at the current time-step. It also has only one set of gain and bias parameters shared over all time-steps.
And more details and development are provided in the complete text.
