I'm applying a single layer LSTM with hidden_size=16 towards a binary classification task. My training and validation loss are both reasonable until around epoch 400 when my learning rate gets halved, and then the two losses start diverging at an extreme rate. What could cause this? Each input is a 20x3 matrix, and I have 194,160 training examples and 49,835 validation examples. I apply the LSTM, then flatten the output into a 20*16=320 length vector, to which I apply a sigmoid output layer.

enter image description here

EDIT: With hidden_size=2, overfitting still occurs but is much more manageable. Also, in terms of validation loss the model performs better than with hidden_size=16. Is it possible that hidden_size=16 is simply too high capacity for my dataset?

enter image description here

  • $\begingroup$ @lerner I edited the question with some more details about the problem setup. To answer your question, num_hidden=16 and time_steps=20. $\endgroup$ – tmakino Oct 20 '17 at 1:44
  • $\begingroup$ Oh, you should not change the num_hidden, you can just project that dimension of each sample in a batch from 16 to a 2 dimensional output. $\endgroup$ – Lerner Zhang Oct 25 '17 at 1:51
  • $\begingroup$ regularization $\endgroup$ – Sycorax says Reinstate Monica Apr 27 '19 at 14:58

I apply the LSTM, then flatten the output into a 20*16=320 length vector, to which I apply a sigmoid output layer.

You should make a decoding process or just a projection projecting the matrix from the 20 * 16 to 20 * 2 with 20 logits(or distributions after apllying a sigmoid layer). Then you can get the entropy(or just logits). .

I mean the sigmoid output layer should be applied to a 20 * 2 matrix you get after the projection, to get the distribution(or you can directly use the logits without apllying sigmoid if you can use TensorFlow's softmax_cross_entropy_with_logits function).

I think this may where the problem lies, and hope it helps.

  • 1
    $\begingroup$ I agree that it's usually the case that LSTM models for classification have a dense layer after the LSTM units, but can you explain how the lack of a dense layer would lead to this behavior? It's not clear to me know the overfitting follows from the (unusual) architecture. $\endgroup$ – Sycorax says Reinstate Monica Sep 10 '18 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.