This is a common phenomenon with RNNs: they can be prone to taking a large step and getting trapped in a virtually flat place in the loss surface. Gradient clipping is successful in mitigating this.

There are at least two ways to decide where to clip the gradients.

1. Track the gradient magnitudes at each iteration and use that information to observe when the gradient becomes too large and the RNN "jumps" to its flat place.
2. Clip at 1.0 and if that's too large or too small, make adjustments from there.

Using a smaller step size might not help because

• the gradient can grow without bound, so choosing a step size that's small enough might be impossible.
• using a small step size everywhere might be undesirable because then your progress towards a minimum is that much slower everywhere.

By contrast, gradient clipping slows your progress only when gradients are too large, but proceeds as normal when they're small enough.

This is a common phenomenon with RNNs: they can be prone to taking a large step and getting trapped in a virtually flat place in the loss surface. Gradient clipping is successful in mitigating this.

There are at least two ways to decide where to clip the gradients.

1. Track the gradient magnitudes at each iteration and use that information to observe when the gradient becomes too large and the RNN "jumps" to its flat place.
2. Choose some value (like 10.0 or 1.0) and if that's too large (still jumps to a flat spot and get stuck) or too small (training is too slow), make adjustments from there.

Using a smaller step size might not help because

• the gradient can grow without bound, so choosing a step size that's small enough might be impossible.
• using a small step size everywhere might be undesirable because then your progress towards a minimum is that much slower everywhere.

By contrast, gradient clipping slows your progress only when gradients are too large, but proceeds as normal when they're small enough.

This is a common phenomenon with RNNs: they can be prone to taking a large step and getting trapped in a virtually flat place in the loss surface. Gradient clipping is successful in mitigating this.

This is a common phenomenon with RNNs: they can be prone to taking a large step and getting trapped in a virtually flat place in the loss surface. Gradient clipping is successful in mitigating this.

There are at least two ways to decide where to clip the gradients.

1. Track the gradient magnitudes at each iteration and use that information to observe when the gradient becomes too large and the RNN "jumps" to its flat place.
2. Clip at 1.0 and if that's too large or too small, make adjustments from there.

Using a smaller step size might not help because

• the gradient can grow without bound, so choosing a step size that's small enough might be impossible.
• using a small step size everywhere might be undesirable because then your progress towards a minimum is that much slower everywhere.

By contrast, gradient clipping slows your progress only when gradients are too large, but proceeds as normal when they're small enough.