I'm applying a single layer LSTM with hidden_size=16 towards a binary classification task. My training and validation loss are both reasonable until around epoch 400 when my learning rate gets halved, and then the two losses start diverging at an extreme rate. What could cause this? Each input is a 20x3 matrix, and I have 194,160 training examples and 49,835 validation examples. I apply the LSTM, then flatten the output into a 20*16=320 length vector, to which I apply a sigmoid output layer.
EDIT: With hidden_size=2, overfitting still occurs but is much more manageable. Also, in terms of validation loss the model performs better than with hidden_size=16. Is it possible that hidden_size=16 is simply too high capacity for my dataset?