I have an Bidirectional LSTM of size 1024, running over a sequences of variable length. My dataset is 100,000 items large, with a 256 batch size. I'm taking the last relevant step, concatenating forward and backward passes, then sending out the results to a fully connected layer.
I've noticed some kind of bouncing that grows bigger the further the training descends. I don't know what this means. I suspect this is happening at the beginning or end of each epoch. In this image I'm around epoch 28.
It seems to be getting worse which is very odd. I'm using the RMSProp optimizer with 0.006 learning rate.
Has anyone seen this sort of thing before?
This seems to be even more pronounced when using the AdamOptimizer. Im applying a small l1_regulariser to the weights of the final output layer. Clearly, the sawtooth look is getting worse