I've got a two conceptual questions about RNNs, particularly LSTMs, which I just can't figure out on my own or with the tutorials I find on the internet. I would really appreciate if you could help me with the following:

  1. If I understand correctly, the states learned within a LSTM are only relevant for one sequence. So, for the next sequence the states are being "relearned" due to $s_{t}=f(Ux_{t} + Ws_{t-1})$ with x being the input at timestep t, s being the state at timestep t and U and W being the matrices that are learned. Is there any good reason why you should use larger batch sizes than 1 with RNNs/LSTMs especially? I know the differences between stoachastic gradient descent, batch gradient descent and Mini-batch gradient descent, but not why the latter two should be preferred over the first one in RNNs/LSTMs.
  2. Why do you need the same sequence lengths within a batch, i.e. why is padding needed? The states are calculated for each sequence separately, so I don't see a reason for this. Does the backprop through time need the same number of states for each sequence, when it's being executed after a batch?
  • 1
    $\begingroup$ Could you clarify question 1? It's not clear to my how your question relates to the rest of the text in the paragraph. It is generally true across all ANN optimization, that using 2 (or more) in a minibatch is more efficient than executing 2 (or more) minibatches of 1 sample each due to highly optimized linear algebra software. $\endgroup$ – Sycorax Jul 4 '17 at 15:31
  • $\begingroup$ @Sycorax Yes, you are right about that, of course. I was actually wondering if there is any connection between LSTMs having a "memory" and using larger batch sizes. So, to rephrase my question: Does it make any difference in LSTMs whether I use batch size 1 or 100 in regard to the states of the cells? There is no special connection for sequences within one batch, except for the fact, that BPTT is being run after a batch? $\endgroup$ – V1nc3nt Jul 4 '17 at 16:54
  • 1
    $\begingroup$ Each sample has its own memory/state -- if you put 100 samples through, the LSTM computes 100 states at each time step. $\endgroup$ – Sycorax Jul 4 '17 at 19:15
  • $\begingroup$ I think you can also consider what type of RNN it is (ex. one to many, many to many, encoder-decoder etc.) For some RNNs the task actually requires that the input and output sequences be aligned, which may influence why padding is used or not $\endgroup$ – information_interchange Aug 13 '18 at 19:21
  1. The state isn't really what is being learned. The weights that determine the state is where the learning happens. The state just holds some abstract representation of what it has seen so far in the sequence, so yes the state is only relevant to the current sequence.
    The advantages of larger batch sizes are better parallelization and smoothing out the gradient so the updates aren't so noisy, and it has no effect on the cells between different training sequences.
  2. You are correct that padding is not necessary and you can operate on sequences of different lengths. But the code is easier to write when it expects all sequences of the same length, so that's usually what you'll see.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.