# Training loss goes down and up again. What is happening?

My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on?

I have two stacked LSTMS as follows (on Keras):

model = Sequential()



I train it for a 100 Epochs:

model.fit(X_train, np.array(y_train), batch_size=1024, nb_epoch=100, validation_split=0.2)


Train on 127803 samples, validate on 31951 samples

And that is what the loss looks like:

• Your learning could be to big after the 25th epoch. Try to set up it smaller and check your loss again – itdxer Mar 11 '16 at 13:17
• But how could extra training make the training data loss bigger? – patapouf_ai Mar 11 '16 at 15:49
• Sorry, I mean learning rate. – itdxer Mar 11 '16 at 15:51
• Thank you itdxer. I think what you said must be on the right track. I tried using "adam" instead of "adadelta" and this solved the problem, though I'm guessing that reducing the learning rate of "adadelta" would probably have worked also. If you want to write a full answer I shall accept it. – patapouf_ai Apr 5 '16 at 15:43

Your learning rate could be to big after the 25th epoch. This problem is easy to identify. You just need to set up a smaller value for your learning rate. If the problem related to your learning rate than NN should reach a lower error despite that it will go up again after a while. The main point is that the error rate will be lower in some point in time.

If you observed this behaviour you could use two simple solutions. First one is a simplest one. Set up a very small step and train it. The second one is to decrease your learning rate monotonically. Here is a simple formula:

$$\alpha(t + 1) = \frac{\alpha(0)}{1 + \frac{t}{m}}$$

Where $a$ is your learning rate, $t$ is your iteration number and $m$ is a coefficient that identifies learning rate decreasing speed. It means that your step will minimise by a factor of two when $t$ is equal to $m$.

• As the OP was using Keras, another option to make slightly more sophisticated learning rate updates would be to use a callback like ReduceLROnPlateau, which reduces the learning rate once the validation loss hasn't improved for a given number of epochs. – n1k31t4 Jan 30 '18 at 15:10