I'm trying to train an RNN, and I'm encountering difficulties with the cost as training progresses. I've had success with my code with previous instances of my code (basically, smaller systems). However, the problem I'm running into is that my training cost initially starts off high, goes down quickly in the first 15 training steps, and then on step 16, the cost increases a lot (but still less than the initial cost). The problem is that the cost doesn't really decrease much after that, even though the training goes on for another 60-ish steps.
Are there any general considerations for such jumps in the cost (which never go down again)? I should specify that the cost does change a bit, but it's a small change. The RNN seems to be getting stuck in some pocket of the state space, unable to improve anymore. Also, I've had a related problem in the past due to exploding gradients (which gives Nan as the cost), but this is slightly different because the cost just gets large (but not infinite). Finally, I do implement learning rate decay during the training, so that shouldn't be an issue (I think). Any tips would be helpful.