# How does LSTM prevent the vanishing gradient problem?

LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from Greff et al.) correspond to the loop around cell.

And I understand that that part can be seen as a sort of identity function, so the derivative is one and the gradient stays constant.

What I don't understand is how it does not vanish due to the other activation functions ? The input, output and forget gates use a sigmoid, which derivative is at most 0.25, and g and h were traditionally tanh. How does backpropagating through those not make the gradient vanish ?

• LSTM is a recurrent neural network model that is very efficient at remembering long term dependencies and that is not vulnerable to the vanishing gradient problem. I am not sure what kind of explanation you are looking for – TheWalkingCube Dec 8 '15 at 12:00
• LSTM : Long Short-Term Memory. (Ref : Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8):1735-80 · December 1997) – horaceT Nov 9 '16 at 23:22
• Gradients in LSTMs do vanish, just slower than in vanilla RNNs, enabling them to catch more distant dependencies. Avoiding the problem of vanishing gradients is still an area of active research. – Artem Sobolev Feb 25 '17 at 7:12
• Care to back the slower vanishing up with a reference? – bayerj Feb 26 '17 at 20:16
• related: quora.com/… – Pinocchio Jun 4 at 0:48

The vanishing gradient is best explained in the one-dimensional case. The multi-dimensional is more complicated but essentially analogous. You can review it in this excellent paper [1].

Assume we have a hidden state $h_t$ at time step $t$. If we make things simple and remove biases and inputs, we have $$h_t = \sigma(w h_{t-1}).$$ Then you can show that

\begin{align} \frac{\partial h_{t'}}{\partial h_t} &= \prod_{k=1}^{t' - t} w \sigma'(w h_{t'-k})\\ &= \underbrace{w^{t' - t}}_{!!!}\prod_{k=1}^{t' - t} \sigma'(w h_{t'-k}) \end{align} The factored marked with !!! is the crucial one. If the weight is not equal to 1, it will either decay to zero exponentially fast in $t'-t$, or grow exponentially fast.

In LSTMs, you have the cell state $s_t$. The derivative there is of the form $$\frac{\partial s_{t'}}{\partial s_t} = \prod_{k=1}^{t' - t} \sigma(v_{t+k}).$$ Here $v_t$ is the input to the forget gate. As you can see, there is no exponentially fast decaying factor involved. Consequently, there is at least one path where the gradient does not vanish. For the complete derivation, see [2].

[1] Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." ICML (3) 28 (2013): 1310-1318.

[2] Bayer, Justin Simon. Learning Sequence Representations. Diss. München, Technische Universität München, Diss., 2015, 2015.

• For lstm, doesn't h_t also depend on h_{t-1}? What do you mean in your paper when you say ds_t/d_s{t-1} "is the only part where the gradients flow through time"? – user3243135 Aug 5 '17 at 22:21
• @user3243135 h_t does depend on h_{t-1}. However, suppose ds_t/d_s{t-1} is kept, even if other gradient flows vanish, the whole gradient flow doesn't vanish. This solves gradient vanishing. – soloice Apr 7 '18 at 10:38
• I always thought that the main issue was the term $$\prod^{t'-t} \sigma'(w h_{t'-k} )$$ because if $\sigma'(z)$ is usually the derivative of a sigmoid (or something with a derivative less than 1) which caused the vanishing gradient for sure (e.g. sigmoids are <1 in magnitude and their derivative is $\sigma'(x) = \sigma(z) (1 - \sigma(z))$ which is <1 for sure ). Is that not why ReLUs became accepted in CNNs? This is one thing that always confused me on the difference in how vanishing gradient was addressed in feed forward models vs Recurrent Models. Any clarifications for this? – Pinocchio Jun 4 at 0:33

The picture of LSTM block from Greff et al. (2015) describes a variant that the authors call vanilla LSTM. It's a bit different from the original definition from Hochreiter & Schmidhuber (1997). The original definition did not include the forget gate and the peephole connections.

The term Constant Error Carousel was used in the original paper to denote the recurrent connection of the cell state. Consider the original definition where the cell state is changed only by addition, when the input gate opens. The gradient of the cell state with regard to the cell state at an earlier time step is zero.

Error may still enter the CEC through the output gate and the activation function. The activation function reduces the magnitude of the error a little bit before it is added to the CEC. CEC is the only place where the error can flow unchanged. Again, when the input gate opens, the error exits through the input gate, activation function, and affine transformation, reducing the magnitude of the error.

Thus the error is reduced when it is backpropagated through an LSTM layer, but only when it enters and exits the CEC. The important thing is that it does not change in the CEC no matter how long distance it travels. This solves the problem in the basic RNN that every time step applies an affine transformation and nonlinearity, meaning that the longer the time distance between the input and output, the smaller the error gets.

http://www.felixgers.de/papers/phd.pdf Please refer to section 2.2 and 3.2.2 where the truncated error part is explained. They don't propagate the error if it leaks out of the cell memory (i.e. if there is a closed/activated input gate), but they update the weights of the gate based on the error only for that time instant. Later it is made zero during further back propagation. This is kind of hack but the reason to do is that the error flow along the gates anyway decay over time.

• Could you expand a little on this? At the moment, the answer will have no value if the link location changes or the paper is taken offline. At the very least it would help to give a full citation (reference) that will allow the paper to be found again if the link stops working, but a short summary that makes this answer self-contained would be best. – Silverfish Jun 27 '16 at 14:22