One of the often cited issues in RNN training is the vanishing gradient problem [1,2,3,4].
However, I came across several papers by Anton Maximilian Schaefer, Steffen Udluft and Hans-Georg Zimmermann (e.g. ) in which they claim that the problem doesn't exist even in a simple RNN, if shared weights are used.
So, which one is true - does the vanishing gradient problem exist or not?
Learning long-term dependencies with gradient descent is difficult by Y.Bengio et al. (1994)
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions by S.Hochreiter (1997)
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies by S.Hochreiter et al. (2003)
On the difficulty of training Recurrent Neural Networks by R.Pascanu et al. (2012)
Learning long-term dependencies with recurrent neural networks by A.M. Schaefer et al. (2008)