In an RNN, if the gradients don't vanish for long/distant terms, won't the derivative of the error be either divergent to infinity or oscillatory?

Question

In my question below, I'll refer to the following well known paper "On the difficulty of training Recurrent Neural Networks" by Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.

Assume we're working on somewhat simplified RNN, where, by their equation (2) on page 1, we have the following dynamical systems of their hidden states $\{x_t\}$:

$x_{t}=W_{R}x_t + W_I \sigma(x_{t-1}) + b$, where I'm using $W_R$ in place of their $W_{rec}$. Let $\theta = (W_R, W_I,b)$ the parameter set of this dynamical system.

Please observe equations (3), (4) and (5) on page 2 that're basically back propagation through time in an unrolled RNN.

If I understand correctly, the gradient vanishing problem is really not the vanishing of the term $\frac{\partial E_t}{\partial \theta}$, but rather vanishing of the terminal summands $\frac{\partial E_t}{\partial x_t}\frac{\partial x_t}{\partial x_k} \frac{\partial x_k}{\partial \theta}$, where $k$ is much small compared to $t$, e.g. for $k=1$. Correct?

If so, I'm having a problem! Assume that the above "vanishing gradient problem", as per the definition above, does not happen. Then the above terminal summand, i.e. $\frac{\partial E_t}{\partial x_t}\frac{\partial x_t}{\partial x_k} \frac{\partial x_k}{\partial \theta}$ does not vanish when $k$ is fixed but $t \to \infty$. But this implies that the sum $\frac{\partial E_t}{\partial \theta}=\Sigma_{k}\frac{\partial E_t}{\partial x_t}\frac{\partial x_t}{\partial x_k} \frac{\partial x_k}{\partial \theta}$ diverges or oscillates as $t \to \infty$. (This should be clear by the contrapositive of the following statement:for a convergent infinite series $\Sigma_{t=0}^{\infty} r_t$, we must have $lim_{t \to \infty} r_t = 0.$).

But if the sum $\frac{\partial E_t}{\partial \theta}$ diverges or oscillates and as $t \to \infty$, then that'd also be a problem right? Because in this case, converging to a local/global minima can be prevented by this divergence or oscillatory behavior of $\frac{\partial E_t}{\partial \theta}$ for large $t$, making "learning" $\theta$ difficult. So why is that not a problem?

Answer 1

There are really 3 cases one might consider:

(1) the longer-term gradients vanish (2) they explode (3) they neither vanish nor explode somehow.

The problem with case (1) is that shorter-term gradients, being added to longer-term gradients, then dominate exponentially the longer-term ones, making it difficult to learn long-term dependencies. Furthermore, in my 1994 paper on the subject, I showed that this condition was a consequence of an RNN which could store memories in a stable way (which is desirable in general).

The problem with case (2) is that SGD breaks down (as it assumes that we make nearly infinitesimal steps, so large gradients tend to make optimization diverge).

Researchers have been striving to achieve case (3), which is not easy, and may come with its own problems (because when you have limited memory, to DO want to forget things which are not relevant, and that implies some form of vanishing gradients).

So indeed, fixing the vanishing gradient problem by obtaining exploding gradients is not a good solution. However, note that some heuristic approach (as we discussed in the Pascanu et al paper cited above) can greatly mitigate the explosion situation (which only happens in rare places during the optimization, at loss cliffs corresponding to boundaries between basins of attraction of the system's dynamics).

Answer 2

Of course we don't want divergence of the gradient. Neither (1) nor (3) are desirable. However, in practice, the sequences are always finite and one can scale down the gradient norm (which is the clipping trick) and avoid the divergence issue for most intents and purposes. Again, you have to realize that the gradient will not be very large all the time, but only in special places where you transition between rather flat parts of the loss landscape, typically. As a result, (3) can be reasonably dealt with, whereas (1) remains much more difficult to handle. Also, it should be noted that as shown in my 1994 paper, condition (1) is actually a consequence of the need to store information (because storing information reliably requires forgetting the perturbations which would otherwise destroy the stored information, and forgetting leads to vanishing gradients). So it looks like there is no completely satisfying solution, but some algorithms may nonetheless work a lot better than others in practice, depending on the type of problem considered.