17
$\begingroup$

It is said that RNN suffers from vanishing gradients when facing long memory conditions, and LSTM is a solution because it enables keeping both long term and short term memories. But I can not understand two basic things:

1- Both outputs of an LSTM cell (cell state and hidden value) are calculated based on previous values of cell state, hidden values and input. Such a recursive operation will make both cell state and hidden variable having long memories. What is the difference?

2- How can we say LSTM reduces chance of vanishing gradients? If gates allow long memory, vanishing gradients will also happen. If they don't and therefore they block long memory chains, how can we say we have long memory operation?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Note that the second question is addressed here stats.stackexchange.com/questions/320919/… $\endgroup$
    – Sycorax - On Strike ♦
    Jun 6, 2018 at 15:48
  • $\begingroup$ Answering first question: Suppose we rename the hidden value as output, based on the fact during multiple layers the hidden value is fed into the layer above and is thus analogous to the output of a neuron in a MLP. Now, the difference between output and cell-state is perhaps easier to understand in a functional sense. The cell-state contains information that is not necessarily useful to output at the current timestep/state, but nevertheless needs to transmitted down in time. $\endgroup$
    – Manoj Kumar
    Feb 2, 2020 at 0:25

2 Answers 2

Reset to default
9
$\begingroup$

Regarding question (2), vanishing/exploding gradients happen in LSTMs too.

In vanilla RNNs, the gradient is a term that depends on a factor exponentiated to $T$ ($T$ is the number of steps you perform backpropagation) [1]. This means that values greater than 1 explode and values less than 1 shrink very fast.

On the other hand, gradients in LSTMs, do not have a term that is exponentiated to $T$ [2]. Therefore, the gradient still shrinks/explodes, but at a lower rate than vanilla RNNs.

[1]: Pascanu, R., Mikolov, T., Bengio, Y., On the difficulty of training Recurrent Neural Networks, Feb. 2013 - https://arxiv.org/pdf/1211.5063.pdf

[2]: Bayer, Justin Simon. Learning Sequence Representations. Diss. München, Technische Universität München, Diss., 2015, 2015 - mentioned in https://stats.stackexchange.com/a/263956/191233

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ please add references for your links in case they die $\endgroup$
    – Antoine
    Jun 6, 2018 at 15:52
  • $\begingroup$ Just for future readers, the T in the paper is not an exponential. It's the transpose of the weight matrix. However, this answer is correct in that W IS raised to a power equal to the total number of time steps in the sequence. This is due to the chain rule when applied to the formulation of the hidden state. Good answer, and it turned me onto a combo of Neural Networks and Dynamical Systems which made me really happy inside. $\endgroup$
    – rocksNwaves
    Jul 30, 2020 at 2:38
  • $\begingroup$ @rocksNwaves Thanks for pointing that out ;) $\endgroup$
    – joaoaccarvalho
    Jul 31, 2020 at 7:41
1
$\begingroup$

Basic units of LSTM networks are LSTM layers that have multiple LSTM cells.

Cells do have internal cell state, often abbreviated as "c", and cells output is what is called a "hidden state", abbreviated as "h".

Regular RNNs do have just the hidden state and no cell state. It turns out that RNNs have difficulty of accessing information from a long time ago.

For instance, check these two long sentences:

Cows, that eat green green ... green grass are OK.

Cow, that eat green green ... green grass is OK.

It would be very hard for RNN to learn the singular/plural dependency, but LSTM are capable of doing that.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.