# Training loss goes down and up again. What is happening?

My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on?

I have two stacked LSTMS as follows (on Keras):

model = Sequential()



I train it for a 100 Epochs:

model.fit(X_train, np.array(y_train), batch_size=1024, nb_epoch=100, validation_split=0.2)


Train on 127803 samples, validate on 31951 samples

And that is what the loss looks like:

• Your learning could be to big after the 25th epoch. Try to set up it smaller and check your loss again – itdxer Mar 11 '16 at 13:17
• But how could extra training make the training data loss bigger? – patapouf_ai Mar 11 '16 at 15:49
• Sorry, I mean learning rate. – itdxer Mar 11 '16 at 15:51
• Thank you itdxer. I think what you said must be on the right track. I tried using "adam" instead of "adadelta" and this solved the problem, though I'm guessing that reducing the learning rate of "adadelta" would probably have worked also. If you want to write a full answer I shall accept it. – patapouf_ai Apr 5 '16 at 15:43

$$\alpha(t + 1) = \frac{\alpha(0)}{1 + \frac{t}{m}}$$
Where $a$ is your learning rate, $t$ is your iteration number and $m$ is a coefficient that identifies learning rate decreasing speed. It means that your step will minimise by a factor of two when $t$ is equal to $m$.