Is there any paper about vanishing-gradients of LSTM?

Question

Some web pages mentioned that LSTM causes the vanishing or exploding gradients if the sequence is too long.

These are one of the pages mention the problem:

• https://machinelearningmastery.com/handle-long-sequences-long-short-term-memory-recurrent-neural-networks/
• https://stackoverflow.com/questions/44478272/how-to-handle-extremely-long-lstm-sequence-length

However, I couldn't find any paper or formulation for it.
Could you please tell me the references for this problem?

Answer 1

Sepp Hochreiter first describes the problem in the 1991 paper The Vanishing Gradient Problem for Recurrent Nets and Solutions. The LSTM is later proposed in the 1997 Long Short-Term Memory by Hochreiter and Schmidhuber.

Answer 2

Although the cell state in the LSTM is separately processed from the hidden layers and only additive updates are done in the cell state preventing gradient vanishing in that path during training, the use of nonlinear activation function in LSTM results in vanishing gradients in other paths than the cell state as mentioned in this paper.

Answer 3

The link below explains (thoroughly enough) how the vanishing gradient issue in LSTM may occur and also provides two sources where specifically in one of them (Bayer,2015) the formulation is discussed. It's the same formulation given in the link. It may be a little difficult to find since the source is a thesis.

Basically it argues that the vanishing gradient issue can still occur in LSTM but, not as easily and as fast as in Vanilla RNN since there exists at least one path to a set of weights that can help prevent the vanishing gradient issue. My opinion is that the argument is a little far-fetched (no one can guarantee that there is such a path in all types of tasks without some serious analysis or mathematical proof).

How does LSTM prevent the vanishing gradient problem?