2
$\begingroup$

Some web pages mentioned that LSTM causes the vanishing or exploding gradients if the sequence is too long.

These are one of the pages mention the problem:

However, I couldn't find any paper or formulation for it.
Could you please tell me the references for this problem?

Improve this question
$\endgroup$
2
  • $\begingroup$ I have not seen any, but it will be good to have a visualization/paper. A hobby project maybe $\endgroup$
    – Anant Gupta
    Jul 17, 2019 at 9:31
  • 2
    $\begingroup$ The whole point of LSTMs is avoiding the vanishing gradient problem. $\endgroup$
    – Jan Kukacka
    Jul 17, 2019 at 10:02

3 Answers 3

Reset to default
3
$\begingroup$

Sepp Hochreiter first describes the problem in the 1991 paper The Vanishing Gradient Problem for Recurrent Nets and Solutions. The LSTM is later proposed in the 1997 Long Short-Term Memory by Hochreiter and Schmidhuber.

Improve this answer
$\endgroup$
1
$\begingroup$

Although the cell state in the LSTM is separately processed from the hidden layers and only additive updates are done in the cell state preventing gradient vanishing in that path during training, the use of nonlinear activation function in LSTM results in vanishing gradients in other paths than the cell state as mentioned in this paper.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is basically a link-only answer (the last sentence) to a question that already has an answer... You might want to expand your answer so that it is at least somewhat self-contained, keeping in mind that the question already has an answer at the linked question. $\endgroup$
    – jbowman
    Nov 29, 2019 at 15:53
  • 1
    $\begingroup$ @jbowman this question is asking about how gradients vanish in LSTM for long sequences, not about how LSTM prevents the gradient vanishing problem arising in a RNN. I have updated my answer by proving a reason of gradient vanishing in LSTM from a paper. Hope that helps. $\endgroup$
    – Md Hishamur Rahman
    Jun 12, 2020 at 16:24
0
$\begingroup$

The link below explains (thoroughly enough) how the vanishing gradient issue in LSTM may occur and also provides two sources where specifically in one of them (Bayer,2015) the formulation is discussed. It's the same formulation given in the link. It may be a little difficult to find since the source is a thesis.

Basically it argues that the vanishing gradient issue can still occur in LSTM but, not as easily and as fast as in Vanilla RNN since there exists at least one path to a set of weights that can help prevent the vanishing gradient issue. My opinion is that the argument is a little far-fetched (no one can guarantee that there is such a path in all types of tasks without some serious analysis or mathematical proof).

How does LSTM prevent the vanishing gradient problem?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.