How do we pass data to a RNN?

Question

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:

Scheme 1

1. Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss)words from each article (for the sake of simplicity let us assume batch size = m)
2. Set the initial hidden state H0 = $[0,0,..,0]$
3. Calculate loss and gradient on this batch and update the parameters
4. Move a word forward and get next S words from each article (hence S2, S3, ... , Ss are the same words as in the previous batch)
5. Use H1 calculated from previous step and use it as H0 in this iteration
6. Do this to the end

*In this scheme we would have to use zero padding on the last batch (at the end of articles)

Scheme 2 Same as Scheme 1 but in step 5 we reinitialize H0 to a vector of Zeroes

Scheme 3 Same as scheme 1 but in step 4 we move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration

Scheme 4, 5, 6 Same as Scheme 1, 2, 3 but instead of taking s consecutive words we take first sentence from each article and zero pad them to the length S

What is the right way to go through the data in feeding it to a RNN, Please give reasons as well. I think it should be Scheme 1 but it could be really slow

Answer 1

Based on how much i got to know i propose the correct scheme would be third one i.e.

1. Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss) words from each article (for the sake of simplicity let us assume batch size = m)
2. Set the initial hidden state H0 = [0,0,..,0]
3. Calculate loss and gradient on this batch and update the parameters
4. We move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration
5. Do this to the end

Why we wont take any but Scheme 1 and Scheme 3 is because: H0 should only be initialized with a vector of zero when there is no context available (if you have some additional information, why not use it?!). The objective is to maximize the probability of each article and the sentences are not independent. Also, there is no reason to not treat period "." as a step in itself. We do calculate its word embedding too

Why not Scheme 1 - Because of catastrophic interference ( As the network in Scheme 1 fed in same input again and again ) combined with slow learning it would result in