1
$\begingroup$

I'm trying to train an LSTM on a regression problem. After observing it for a while, I'm noticing a strange effect. The training loss decreases every epoch, which is expected, however the validation loss swings from dramatic improvement to dramatic worsening.

For example, in epoch 10 (e.g.), the validation loss may drop by 200 (using MAPE). The next epoch, though, it might then increase by 300. It alternates between improving and worsening every couple of epochs.

I'm using Keras to build the network, using ADAM as the optimizer, with a linear activation on the final layer. Here's some sample output:

Epoch 9/20
146000/146068 [============================>.] - ETA: 3s - loss: 134.2498 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00008: saving model to /tmp/146068/146068 [==============================] - 9100s - loss: 134.2351 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 373.6942 - val_acc: 0.0000e+00 - val_fbeta_score: nan
Epoch 10/20
146000/146068 [============================>.] - ETA: 3s - loss: 157.5198 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00009: saving model to /tmp/146068/146068 [==============================] - 9074s - loss: 157.4901 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 214.8772 - val_acc: 0.0000e+00 - val_fbeta_score: nan
Epoch 11/20
146000/146068 [============================>.] - ETA: 3s - loss: 140.6320 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00010: saving model to /tmp/146068/146068 [==============================] - 9088s - loss: 140.6223 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 568.2613 - val_acc: 0.0000e+00 - val_fbeta_score: nan
Epoch 12/20
146000/146068 [============================>.] - ETA: 3s - loss: 151.2053 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00011: saving model to /tmp/146068/146068 [==============================] - 9085s - loss: 151.1793 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 223.2714 - val_acc: 0.0000e+00 - val_fbeta_score: nan
Epoch 13/20
146000/146068 [============================>.] - ETA: 3s - loss: 129.6313 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00012: saving model to /tmp/146068/146068 [==============================] - 9059s - loss: 129.6153 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 264.8294 - val_acc: 0.0000e+00 - val_fbeta_score: nan
Epoch 14/20
146000/146068 [============================>.] - ETA: 3s - loss: 145.4163 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00013: saving model to /tmp/146068/146068 [==============================] - 9058s - loss: 145.4047 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 292.1805 - val_acc: 0.0000e+00 - val_fbeta_score: nan
Epoch 15/20
146000/146068 [============================>.] - ETA: 3s - loss: 131.6223 - acc: 0.0000e+00 - fbeta_score: nan Epoch 00014: saving model to /tmp/146068/146068 [==============================] - 9059s - loss: 131.6051 - acc: 0.0000e+00 - fbeta_score: nan - val_loss: 543.0168 - val_acc: 0.0000e+00 - val_fbeta_score: nan

Does anyone have ideas on what may cause this sort of behavior?

Thanks!

$\endgroup$
  • $\begingroup$ The loss you report is on one batch, one epoch or a moving average over the epoch? If it's only over the batch, then the swings are due to stochasticity $\endgroup$ – Rob Romijnders Dec 29 '16 at 19:44
  • $\begingroup$ I second what Rob said. It could be stochasticity, train longer and plot a chart of the average of the moving average of the loss. If it keeps going down on average, then it's okay. You could also try and tweak parameters. Also give other optimisers a try, especially rmsprop, which often goes well with LSTM. That's all I can say without seeing any code, params or data. $\endgroup$ – Diego Frata Dec 29 '16 at 19:58
  • $\begingroup$ @RobRomijnders thanks for the comment. The loss is one epoch. Do you think training for a longer amount of time would be the right way to go? $\endgroup$ – Shanif Dec 30 '16 at 15:16
  • $\begingroup$ @DiegoFrata thanks for the comment as well, appreciate the insight. I was thinking that tweaking or tuning the network could help. I'll give rmsprop a shot. Really appreciate the insight, thanks so much! $\endgroup$ – Shanif Dec 30 '16 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.