# What happens to the initial hidden state in an RNN layer?

I thought I knew how RNNs work, however, when I tried to actually implement it myself, I faced some issues. For one, how do we deal with the initial hidden state?
At the very beginning we just create a vector of zeros with some length which is then used to create the next hidden state and this goes on until we traverse all timesteps. Now this is for one iteration, What does happen in the next iterations?
When we get new inputs, should we still be feeding the same vector of zeros to the network? this doesn't seem right! since in the backprop stage, it seems we don't update the h0! I'm confused here.
If we always feed the same zero vector, it just nullifies all the previous updates we have done on all hidden states so far! so what is it that needs to be done to the initial state?

• You can just learn the initial hidden state. So it should be randomly initialized like all the other weights and updated during backpropagation. – user0 Mar 3 '19 at 16:36
• There are two common RNN strategies. (1) You have a long sequence that's always contiguous (for example, a language model that's trained on the text of War and Peace); because the novel's words all have a very specific order, you have to train it on consecutive sequences, so the hidden state at the last hidden state of the previous sequence is used as the initial hidden state of the next sequence. (2) You have lots of related, but distinct sequences (such as discrete tweets); it can make sense to start each sequence with hidden states of all 0s. Which applies to you? – Sycorax Mar 3 '19 at 17:32
• Thank you very much guys I really appreciate it. @Sycorax : it makes sense now! thanks a lot :) so both strategies are OK! I dont have any examples at the moment, I'm just trying to implement one, so that I can run and play with different examples when its finished. – Rika Mar 3 '19 at 18:11

There are two common RNN strategies.

1. You have a long sequence that's always contiguous (for example, a language model that's trained on the text of War and Peace); because the novel's words all have a very specific order, you have to train it on consecutive sequences, so the hidden state at the last hidden state of the previous sequence is used as the initial hidden state of the next sequence.

The way most people do this is that you'll have to traverse the sequences in order, and not shuffle. Suppose you use mini-batch size of 2. You can cut the book in half, and the first sample will always have text from the first half of War and Peace and the second sample will always have text from the second half. Instead of using samples at random, the text is always read in order, so the first sample in the first mini-batch has the first words of the text, and the second sample in the first mini-batch has the first words after the mid-point of the text.

Purely abstractly, I suppose you could do something more complicated where you shuffle the data but can compute the initial hidden state for each position in the sequence (e.g. by computing the text up until that point, or else saving & restoring states) but this sounds expensive.

2. You have lots of distinct sequences (such as discrete tweets); it can make sense to start each sequence with hidden states of all 0s. Some people prefer to train a "baseline" initial state (user0's suggestion). I read an article advocating doing this if your data has lots of short sequences but I can't find the article now.

Which strategy is appropriate depends on the problem, and specific choices about how to represent that problem.

From the perspective of developing software, an ideal implementation would somehow expose functionality for both options to users. This can be tricky, and different software (pytorch, tensorflow, keras) achieves this in different ways.

• In the second scenario, each sequence (e.g. tweet) its its own atomic item, unrelated to the items before and after, so every item starts with an initial state that's 0s. – Sycorax Mar 3 '19 at 19:22
• @Breeze The way most people solve this is that you'll have to traverse the sequences in order, and not shuffle. Suppose you use minibatch size of 2. You can cut the book in half, and the first sample will always have text from the first half of War and Peace and the second sample will always have text from the second half. Instead of using samples at random, the text is always read in order, so the first sample in the first minibatch has the first words of the text, and the second sample in the first minibatch has the first words after the mid-point of the text. – Sycorax Mar 3 '19 at 19:52
• Purely abstractly, I suppose you could do something more complicated where you shuffle the data but can compute the initial hidden state for each position in the sequence (e.g. by computing the text up until that point, or else saving & restoring states) but this sounds expensive. – Sycorax Mar 3 '19 at 19:55
• Thanks a gazillion times :) I get it now. God bless you :) – Rika Mar 3 '19 at 19:56
• All of this knowledge was garnered because I wanted to make a Twitter bot to automatically generate almost-intelligible tweets. – Sycorax Mar 3 '19 at 19:57