Why does the training cost of my model decrease, then suddenly increases, never going down again?

Question

I'm trying to train an RNN, and I'm encountering difficulties with the cost as training progresses. I've had success with my code with previous instances of my code (basically, smaller systems). However, the problem I'm running into is that my training cost initially starts off high, goes down quickly in the first 15 training steps, and then on step 16, the cost increases a lot (but still less than the initial cost). The problem is that the cost doesn't really decrease much after that, even though the training goes on for another 60-ish steps.

Are there any general considerations for such jumps in the cost (which never go down again)? I should specify that the cost does change a bit, but it's a small change. The RNN seems to be getting stuck in some pocket of the state space, unable to improve anymore. Also, I've had a related problem in the past due to exploding gradients (which gives Nan as the cost), but this is slightly different because the cost just gets large (but not infinite). Finally, I do implement learning rate decay during the training, so that shouldn't be an issue (I think). Any tips would be helpful.

Answer 1

This is a common phenomenon with RNNs: they can be prone to taking a large step and getting trapped in a virtually flat place in the loss surface. Gradient clipping is successful in mitigating this.

There are at least two ways to decide where to clip the gradients.

1. Track the gradient magnitudes at each iteration and use that information to observe when the gradient becomes too large and the RNN "jumps" to its flat place.
2. Choose some value (like 10.0 or 1.0) and if that's too large (still jumps to a flat spot and get stuck) or too small (training is too slow), make adjustments from there.

Using a smaller step size might not help because

• the gradient can grow without bound, so choosing a step size that's small enough might be impossible.
• using a small step size everywhere might be undesirable because then your progress towards a minimum is that much slower everywhere.

By contrast, gradient clipping slows your progress only when gradients are too large, but proceeds as normal when they're small enough.