# Meaning of batch sizes for RNNs/LSTMs and reasons for padding

I've got a two conceptual questions about RNNs, particularly LSTMs, which I just can't figure out on my own or with the tutorials I find on the internet. I would really appreciate if you could help me with the following:

1. If I understand correctly, the states learned within a LSTM are only relevant for one sequence. So, for the next sequence the states are being "relearned" due to $$s_{t}=f(Ux_{t} + Ws_{t-1})$$ with x being the input at timestep t, s being the state at timestep t and U and W being the matrices that are learned. Is there any good reason why you should use larger batch sizes than 1 with RNNs/LSTMs especially? I know the differences between stoachastic gradient descent, batch gradient descent and Mini-batch gradient descent, but not why the latter two should be preferred over the first one in RNNs/LSTMs.
2. Why do you need the same sequence lengths within a batch, i.e. why is padding needed? The states are calculated for each sequence separately, so I don't see a reason for this. Does the backprop through time need the same number of states for each sequence, when it's being executed after a batch?
• Could you clarify question 1? It's not clear to my how your question relates to the rest of the text in the paragraph. It is generally true across all ANN optimization, that using 2 (or more) in a minibatch is more efficient than executing 2 (or more) minibatches of 1 sample each due to highly optimized linear algebra software. – Sycorax Jul 4 '17 at 15:31
• @Sycorax Yes, you are right about that, of course. I was actually wondering if there is any connection between LSTMs having a "memory" and using larger batch sizes. So, to rephrase my question: Does it make any difference in LSTMs whether I use batch size 1 or 100 in regard to the states of the cells? There is no special connection for sequences within one batch, except for the fact, that BPTT is being run after a batch? – V1nc3nt Jul 4 '17 at 16:54
• Each sample has its own memory/state -- if you put 100 samples through, the LSTM computes 100 states at each time step. – Sycorax Jul 4 '17 at 19:15
• I think you can also consider what type of RNN it is (ex. one to many, many to many, encoder-decoder etc.) For some RNNs the task actually requires that the input and output sequences be aligned, which may influence why padding is used or not – information_interchange Aug 13 '18 at 19:21