Best way to initialize LSTM state I was wondering what is the best way to initialize the state for LSTMs.  Currently I just initialize it to all zeros.  I can not really find anything online about how to initialize it.  One thing I was thinking of doing is making the initial state a trainable parameter.  Any advice?
 A: Normally, you would set the initial states to zero, but the network is going to learn to adapt to that initial state.
The following article suggests learning the initial hidden states or using random noise.
Basically, if your data includes many short sequences, then training the initial state can accelerate learning.
Alternatively, if your data includes a small number of long sequences then there may not be enough data to effectively train the initial state. In that case using a noisy initial state can accelerate learning. An idea they don't mention would be to learn the mean and std of the noise generator.
The article notes that if you choose to learn the initial state, then adding noise is of little benefit.
A: You can use initialized parameters that are learned using transfer learning, but keep in mind that it also began somewhere from a non-learned initialized state. Basically, you have to start from some point, usually a bunch of zeros, and then refine by training. So, if you are not using any transfer learning mechanisms, you also have to start from a manual initial state, I am sure there might be works of literature available for manually setting the initial states.
This is the simplest explanation I could put. Thank You.
