It is said that RNN suffers from vanishing gradients when facing long memory conditions, and LSTM is a solution because it enables keeping both long term and short term memories. But I can not understand two basic things:

1- Both outputs of an LSTM cell (cell state and hidden value) are calculated based on previous values of cell state, hidden values and input. Such a recursive operation will make both cell state and hidden variable having long memories. What is the difference?

2- How can we say LSTM reduces chance of vanishing gradients? If gates allow long memory, vanishing gradients will also happen. If they don't and therefore they block long memory chains, how can we say we have long memory operation?


Regarding question (2), vanishing/exploding gradients happen in LSTMs too.

In vanilla RNNs, the gradient is a term that depends on a factor exponentiated to $T$ ($T$ is the number of steps you perform backpropagation) [1]. This means that values greater than 1 explode and values less than 1 shrink very fast.

On the other hand, gradients in LSTMs, do not have a term that is exponentiated to $T$ [2]. Therefore, the gradient still shrinks/explodes, but at a lower rate than vanilla RNNs.

[1]: Pascanu, R., Mikolov, T., Bengio, Y., On the difficulty of training Recurrent Neural Networks, Feb. 2013 - https://arxiv.org/pdf/1211.5063.pdf

[2]: Bayer, Justin Simon. Learning Sequence Representations. Diss. München, Technische Universität München, Diss., 2015, 2015 - mentioned in https://stats.stackexchange.com/a/263956/191233

  • 1
    $\begingroup$ please add references for your links in case they die $\endgroup$ – Antoine Jun 6 '18 at 15:52

Basic units of LSTM networks are LSTM layers that have multiple LSTM cells.

Cells do have internal cell state, often abbreviated as "c", and cells output is what is called a "hidden state", abbreviated as "h".

Regular RNNs do have just the hidden state and no cell state. It turns out that RNNs have difficulty of accessing information from a long time ago.

For instance, check these two long sentences:

Cows, that eat green green ... green grass are OK.

Cow, that eat green green ... green grass is OK.

It would be very hard for RNN to learn the singular/plural dependency, but LSTM are capable of doing that.


Not the answer you're looking for? Browse other questions tagged or ask your own question.