# How does LSTM prevent the vanishing gradient problem?

LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from Greff et al.) correspond to the loop around cell.

And I understand that that part can be seen as a sort of identity function, so the derivative is one and the gradient stays constant.

What I don't understand is how it does not vanish due to the other activation functions ? The input, output and forget gates use a sigmoid, which derivative is at most 0.25, and g and h were traditionally tanh. How does backpropagating through those not make the gradient vanish ?

• LSTM is a recurrent neural network model that is very efficient at remembering long term dependencies and that is not vulnerable to the vanishing gradient problem. I am not sure what kind of explanation you are looking for Dec 8, 2015 at 12:00
• LSTM : Long Short-Term Memory. (Ref : Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8):1735-80 · December 1997) Nov 9, 2016 at 23:22
• Gradients in LSTMs do vanish, just slower than in vanilla RNNs, enabling them to catch more distant dependencies. Avoiding the problem of vanishing gradients is still an area of active research. Feb 25, 2017 at 7:12
• Care to back the slower vanishing up with a reference? Feb 26, 2017 at 20:16
• related: quora.com/… Jun 4, 2019 at 0:48

The vanishing gradient is best explained in the one-dimensional case. The multi-dimensional is more complicated but essentially analogous. You can review it in this excellent paper .

Assume we have a hidden state $h_t$ at time step $t$. If we make things simple and remove biases and inputs, we have $$h_t = \sigma(w h_{t-1}).$$ Then you can show that

\begin{align} \frac{\partial h_{t'}}{\partial h_t} &= \prod_{k=1}^{t' - t} w \sigma'(w h_{t'-k})\\ &= \underbrace{w^{t' - t}}_{!!!}\prod_{k=1}^{t' - t} \sigma'(w h_{t'-k}) \end{align} The factored marked with !!! is the crucial one. If the weight is not equal to 1, it will either decay to zero exponentially fast in $t'-t$, or grow exponentially fast.

In LSTMs, you have the cell state $s_t$. The derivative there is of the form $$\frac{\partial s_{t'}}{\partial s_t} = \prod_{k=1}^{t' - t} \sigma(v_{t+k}).$$ Here $v_t$ is the input to the forget gate. As you can see, there is no exponentially fast decaying factor involved. Consequently, there is at least one path where the gradient does not vanish. For the complete derivation, see .

 Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." ICML (3) 28 (2013): 1310-1318.

 Bayer, Justin Simon. Learning Sequence Representations. Diss. München, Technische Universität München, Diss., 2015, 2015.

• For lstm, doesn't h_t also depend on h_{t-1}? What do you mean in your paper when you say ds_t/d_s{t-1} "is the only part where the gradients flow through time"? Aug 5, 2017 at 22:21
• @user3243135 h_t does depend on h_{t-1}. However, suppose ds_t/d_s{t-1} is kept, even if other gradient flows vanish, the whole gradient flow doesn't vanish. This solves gradient vanishing. Apr 7, 2018 at 10:38
• I always thought that the main issue was the term $$\prod^{t'-t} \sigma'(w h_{t'-k} )$$ because if $\sigma'(z)$ is usually the derivative of a sigmoid (or something with a derivative less than 1) which caused the vanishing gradient for sure (e.g. sigmoids are <1 in magnitude and their derivative is $\sigma'(x) = \sigma(z) (1 - \sigma(z))$ which is <1 for sure ). Is that not why ReLUs became accepted in CNNs? This is one thing that always confused me on the difference in how vanishing gradient was addressed in feed forward models vs Recurrent Models. Any clarifications for this? Jun 4, 2019 at 0:33
• The gradient of the sigmoid could also become a problem, assuming a distribution of inputs with large variance and/or mean away from 0. However, even if you use ReLUs, the main problem persists: multiplying repeatedly by a matrix of weights (usually small) causes vanishing gradients, or in some cases, where regularization has not been adequate, exploding gradients. Oct 8, 2019 at 7:12
• Hi as you mentioned that in LSTM vt is input to the forget gate which is basically output of the previous cell, right? And out of previous cell is combination of forget gate and input gate where both of them have some w assigned to them. Then how're we independent of w in derivative of LSTM. Sorry if you feel it's stupid question Nov 10, 2021 at 9:13

I'd like to add some detail to the accepted answer, because I think it's a bit more nuanced and the nuance may not be obvious to someone first learning about RNNs.

For the vanilla RNN, $$\frac{\partial h_{t'}}{\partial h_{t}} = \prod _{k=1} ^{t'-t} w \sigma'(w h_{t'-k})$$.

For the LSTM, $$\frac{\partial s_{t'}}{\partial s_{t}} = \prod _{k=1} ^{t'-t} \sigma(v_{t+k})$$

• a natural question to ask is, don't both the product-sums have a sigmoid term that when multiplied together $$t'-t$$ times can vanish?
• the answer is yes, which is why LSTM will suffer from vanishing gradients as well, but not nearly as much as the vanilla RNN

The difference is for the vanilla RNN, the gradient decays with $$w \sigma'(\cdot)$$ while for the LSTM the gradient decays with $$\sigma (\cdot)$$.

For the LSTM, there's is a set of weights which can be learned such that $$\sigma (\cdot) \approx 1$$ Suppose $$v_{t+k} = wx$$ for some weight $$w$$ and input $$x$$. Then the neural network can learn a large $$w$$ to prevent gradients from vanishing.

e.g. In the 1D case if $$x=1$$, $$w=10$$ $$v_{t+k}=10$$ then the decay factor $$\sigma (\cdot) = 0.99995$$, or the gradient dies as: $$(0.99995)^{t'-t}$$

For the vanilla RNN, there is no set of weights which can be learned such that $$w \sigma'(w h_{t'-k}) \approx 1$$

e.g. In the 1D case, suppose $$h_{t'-k}=1$$. The function $$w \sigma'(w*1)$$ achieves a maximum of $$0.224$$ at $$w=1.5434$$. This means the gradient will decay as, $$(0.224)^{t'-t}$$

• For time t to t', the product is written as from 1 to t'-t. Is that a Markov property? Oct 20, 2021 at 1:27
• What if we use ReLU as activation function? Jun 23 at 19:48

http://www.felixgers.de/papers/phd.pdf Please refer to section 2.2 and 3.2.2 where the truncated error part is explained. They don't propagate the error if it leaks out of the cell memory (i.e. if there is a closed/activated input gate), but they update the weights of the gate based on the error only for that time instant. Later it is made zero during further back propagation. This is kind of hack but the reason to do is that the error flow along the gates anyway decay over time.

• Could you expand a little on this? At the moment, the answer will have no value if the link location changes or the paper is taken offline. At the very least it would help to give a full citation (reference) that will allow the paper to be found again if the link stops working, but a short summary that makes this answer self-contained would be best. Jun 27, 2016 at 14:22

The picture of LSTM block from Greff et al. (2015) describes a variant that the authors call vanilla LSTM. It's a bit different from the original definition from Hochreiter & Schmidhuber (1997). The original definition did not include the forget gate and the peephole connections.

The term Constant Error Carousel was used in the original paper to denote the recurrent connection of the cell state. Consider the original definition where the cell state is changed only by addition, when the input gate opens. The gradient of the cell state with regard to the cell state at an earlier time step is zero.

Error may still enter the CEC through the output gate and the activation function. The activation function reduces the magnitude of the error a little bit before it is added to the CEC. CEC is the only place where the error can flow unchanged. Again, when the input gate opens, the error exits through the input gate, activation function, and affine transformation, reducing the magnitude of the error.

Thus the error is reduced when it is backpropagated through an LSTM layer, but only when it enters and exits the CEC. The important thing is that it does not change in the CEC no matter how long distance it travels. This solves the problem in the basic RNN that every time step applies an affine transformation and nonlinearity, meaning that the longer the time distance between the input and output, the smaller the error gets.