My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on?

I have two stacked LSTMS as follows (on Keras):

model = Sequential()
model.add(LSTM(512, return_sequences=True, input_shape=(len(X[0]), len(nd.char_indices))))
model.add(LSTM(512, return_sequences=False))

model.compile(loss='binary_crossentropy', optimizer='adadelta')

I train it for a 100 Epochs:

model.fit(X_train, np.array(y_train), batch_size=1024, nb_epoch=100, validation_split=0.2)

Train on 127803 samples, validate on 31951 samples

And that is what the loss looks like: Graph of Loss

  • 2
    $\begingroup$ Your learning could be to big after the 25th epoch. Try to set up it smaller and check your loss again $\endgroup$
    – itdxer
    Mar 11, 2016 at 13:17
  • $\begingroup$ But how could extra training make the training data loss bigger? $\endgroup$ Mar 11, 2016 at 15:49
  • 3
    $\begingroup$ Sorry, I mean learning rate. $\endgroup$
    – itdxer
    Mar 11, 2016 at 15:51
  • $\begingroup$ Thank you itdxer. I think what you said must be on the right track. I tried using "adam" instead of "adadelta" and this solved the problem, though I'm guessing that reducing the learning rate of "adadelta" would probably have worked also. If you want to write a full answer I shall accept it. $\endgroup$ Apr 5, 2016 at 15:43

1 Answer 1


Your learning rate could be to big after the 25th epoch. This problem is easy to identify. You just need to set up a smaller value for your learning rate. If the problem related to your learning rate than NN should reach a lower error despite that it will go up again after a while. The main point is that the error rate will be lower in some point in time.

If you observed this behaviour you could use two simple solutions. First one is a simplest one. Set up a very small step and train it. The second one is to decrease your learning rate monotonically. Here is a simple formula:

$$ \alpha(t + 1) = \frac{\alpha(0)}{1 + \frac{t}{m}} $$

Where $a$ is your learning rate, $t$ is your iteration number and $m$ is a coefficient that identifies learning rate decreasing speed. It means that your step will minimise by a factor of two when $t$ is equal to $m$.

  • 9
    $\begingroup$ As the OP was using Keras, another option to make slightly more sophisticated learning rate updates would be to use a callback like ReduceLROnPlateau, which reduces the learning rate once the validation loss hasn't improved for a given number of epochs. $\endgroup$
    – n1k31t4
    Jan 30, 2018 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.