One of the often cited issues in RNN training is the vanishing gradient problem [1,2,3,4].

However, I came across several papers by Anton Maximilian Schaefer, Steffen Udluft and Hans-Georg Zimmermann (e.g. [5]) in which they claim that the problem doesn't exist even in a simple RNN, if shared weights are used.

So, which one is true - does the vanishing gradient problem exist or not?

  1. Learning long-term dependencies with gradient descent is difficult by Y.Bengio et al. (1994)

  2. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions by S.Hochreiter (1997)

  3. Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies by S.Hochreiter et al. (2003)

  4. On the difficulty of training Recurrent Neural Networks by R.Pascanu et al. (2012)

  5. Learning long-term dependencies with recurrent neural networks by A.M. Schaefer et al. (2008)

  • 4
    $\begingroup$ Cross-posted to Computer Science, Theoretical Computer Science, Cross Validated and Stack Overflow! Please do not do this. Cross-posting fragments answers and wastes people's time when the answer a question that already has a good answer somewhere else that they've not seen. $\endgroup$ Jul 26, 2014 at 8:07
  • $\begingroup$ Note that the version of Pascanu's paper that you linked to is not the most up to date one. This is the most up to date version (and this is actually the only version that appears when you search for the paper's name in google scholar) $\endgroup$ Sep 29, 2018 at 6:04
  • $\begingroup$ Also, I think it would really help if you included a quote of the main idea of the contrarian paper (Schaefer et al), e.g. "We noticed that under certain conditions vanishing gradients do indeed occur, but are only a problem if we put a static view on the networks like it has been done in [5,6]. Studying the development of the error flow during the learning process we observed that the networks themselves have a regularising effect, i.e., they are able to prolong their information flow and consequently solve the problem of a vanishing gradient." $\endgroup$ Sep 29, 2018 at 6:19

1 Answer 1


First let's restate the problem of vanishing gradients. Suppose you have a normal multilayer perceptron with sigmoidal hidden units. This is trained by back-propagation. When there are many hidden layers the error gradient weakens as it moves from the back of the network to the front, because the derivative the sigmoid weakens towards the poles. The updates as you move to the front of the network will contain less information.

RNNs amplify this problem because they are trained by back-propagation through time (BPTT). Effectively the number of layers that is traversed by back-propagation grows dramatically.

The long short term memory (LSTM) architecture to avoids the problem of vanishing gradients by introducing error gating. This allows it to learn long term (100+ step) dependencies between data points through "error carousels."

A more recent trend in training neural networks is to use rectified linear units, which are more robust towards the vanishing gradient problem. RNNs with sparsity penalization and rectified linear unit apparently work well.

See Advances In Optimizing Recurrent Networks.

Historically neural networks performance greatly depended on many optimization tricks and the selection of many hyperparameters. In the case of RNN you'd be wise to also implement rmsprop and Nesterov’s accelerated gradient. Thankfully, the recent developments in dropout training have made neural networks more robust towards overfitting. Apparently there is some work towards making dropout work with RNNs.

See On Fast Dropout and its Applicability to Recurrent Networks

  • $\begingroup$ Actually, the fifth paper does not use LSTM, but shows that even for simple RNN the gradient does not vanish for 100 time steps. Hence the question. $\endgroup$
    – qwer1304
    Jul 26, 2014 at 16:55
  • $\begingroup$ I edited my response to reflect that. Sorry, I do not have access to that paper so I cannot comment if they are doing anything besides Elman RNN. AFAIK, The vanishing/exploding gradient problem of RNNs is explained analytically and not empirically. The possible cures are LSTM, Hessian Free optimization, echo state networks, smart weight initialization, rectifier with sparsity, NAD, rmsprop, et al. Presumably the linked paper promotes one or several of those. $\endgroup$ Jul 26, 2014 at 17:53
  • $\begingroup$ There seem to be solutions with either the parameterization (tricks on gradients) or optimization (SGD variants) perspective. Are you aware of papers and texts that discussing this in a structured manner? $\endgroup$
    – siegfried
    Oct 1, 2021 at 0:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.