I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow of values.
However, apparently, I haven't fully understood how LSTM solves the "vanishing and exploding gradients" problem, which occurs while training, using back-propagation through time, a conventional RNN. I haven't had the opportunity to read the papers to fully understand the math.
This answer gives a brief explanation of how RNNs with LSTM units solve the "vanishing gradients" problem. Mathematically, the reason seems to be the inexistence of a derivative which does not vanish, i.e. does not tend to zero. Consequently, the author states, "there is at least one path where the gradient does not vanish". IMHO, this explanation is a bit vague.
Meanwhile, I was reading the paper Sequence to Sequence Learning with Neural Networks (by Ilya Sutskever, Oriol Vinyals, Quoc V. Le), and, in that paper, section "3.4 Training details", it is stated
Although LSTMs tend to not suffer from the vanishing gradient problem, they can have exploding gradients.
I have always thought that RNNs with LSTM units solve both the "vanishing" and "exploding gradients" problems, but, apparently, RNNs with LSTM units also suffer from "exploding gradients".
Intuitively, why is that? Mathematically, what are the reasons?