I'm trying to train an RNN, and I'm encountering difficulties with the cost as training progresses. I've had success with my code with previous instances of my code (basically, smaller systems). However, the problem I'm running into is that my training cost initially starts off high, goes down quickly in the first 15 training steps, and then on step 16, the cost increases a lot (but still less than the initial cost). The problem is that the cost doesn't really decrease much after that, even though the training goes on for another 60-ish steps.

Are there any general considerations for such jumps in the cost (which never go down again)? I should specify that the cost does change a bit, but it's a small change. The RNN seems to be getting stuck in some pocket of the state space, unable to improve anymore. Also, I've had a related problem in the past due to exploding gradients (which gives Nan as the cost), but this is slightly different because the cost just gets large (but not infinite). Finally, I do implement learning rate decay during the training, so that shouldn't be an issue (I think). Any tips would be helpful.


This is a common phenomenon with RNNs: they can be prone to taking a large step and getting trapped in a virtually flat place in the loss surface. Gradient clipping is successful in mitigating this.

There are at least two ways to decide where to clip the gradients.

  1. Track the gradient magnitudes at each iteration and use that information to observe when the gradient becomes too large and the RNN "jumps" to its flat place.
  2. Choose some value (like 10.0 or 1.0) and if that's too large (still jumps to a flat spot and get stuck) or too small (training is too slow), make adjustments from there.

Using a smaller step size might not help because

  • the gradient can grow without bound, so choosing a step size that's small enough might be impossible.
  • using a small step size everywhere might be undesirable because then your progress towards a minimum is that much slower everywhere.

By contrast, gradient clipping slows your progress only when gradients are too large, but proceeds as normal when they're small enough.

| cite | improve this answer | |
  • $\begingroup$ Two questions. First, is there a good way to determine where to clip the gradients (my understanding is that you cap them to a certain size)? And second, would simply starting with a smaller step size help? $\endgroup$ – Germ Sep 12 at 20:43
  • $\begingroup$ @Germ I've edited my answer. $\endgroup$ – Sycorax Sep 12 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.