I have an Bidirectional LSTM of size 1024, running over a sequences of variable length. My dataset is 100,000 items large, with a 256 batch size. I'm taking the last relevant step, concatenating forward and backward passes, then sending out the results to a fully connected layer.

I've noticed some kind of bouncing that grows bigger the further the training descends. I don't know what this means. I suspect this is happening at the beginning or end of each epoch. In this image I'm around epoch 28.

bouncing training error

It seems to be getting worse which is very odd. I'm using the RMSProp optimizer with 0.006 learning rate.

Has anyone seen this sort of thing before?

Cheers B


This seems to be even more pronounced when using the AdamOptimizer. Im applying a small l1_regulariser to the weights of the final output layer. Clearly, the sawtooth look is getting worse

adam version

  • $\begingroup$ Is your learning rate decaying? $\endgroup$ – Alex R. May 31 '18 at 20:37
  • $\begingroup$ Im using tf.train.RMSPropOptimizer which I believe has a 0.9 decay by default. $\endgroup$ – Oni May 31 '18 at 21:07
  • $\begingroup$ The “bounces” look really similar. My guess is that it’s overfitting and bouncing because of outliers;. found some minima, but some data points are outliers and pushes it up, and then it goes pack down when others data comes. Repeats in epochs? $\endgroup$ – Andreas Storvik Strauman May 31 '18 at 22:34
  • $\begingroup$ I think you might be right, given a few related issues. Overfitting and memorising does seem likely. Thanks. $\endgroup$ – Oni Jun 1 '18 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.