On a binary classification task, I can get perfect performance on the training set, but no matter how strongly I regularize the recurrent neural network (dropout 0.99, L2 weight penalties of 0.01) the generalization performance on a validation set is poor. What strikes me as unusual, is that the accuracy on the validation set always seem to get stuck at the same value of 76%. No matter which architecture or regularization alternative, this is a threshold that my model does not seem to be able to overcome.

Can this be a general local minima that is being reached, although the training set performance is pretty much perfect? (loss ~0, accuracy 99%)

Below one example of several similar looking ones on training/validation loss and accuracy: Note that a plateau is reached around 0.7, after which the training set performance (blue) jumps while the validation set (red) gets worse again (overfitting).

enter image description here

On the network: For the image, a single layer RNN is used, with 100 LSTM units. Targets are binary labels {0,1}, class balanced. Loss function is the softmax_cross_entropy_with_logits, using Adam as optimization algorithm.


model = Sequential()
model.add(Bidirectional(LSTM(800, return_sequences=False),
                    input_shape=(window_size, num_input)))
adam = Adam(lr=5e-6, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
model.compile(loss='categorical_crossentropy', optimizer=adam,metrics=['accuracy'])

Training with Bidirectional LSTM in Keras. Hidden_Units = 200, Dropout = 0.95 Training with Bidirectional LSTM in Keras. Hidden_Units = 200, Dropout = 0.95

  • $\begingroup$ Loss decreasing on training data and increasing on validation data is classic overfitting. Minima obtained by Adam tend to be sharper than those obtained by momentum or vanilla SGD; perhaps try another optimizer? $\endgroup$ – Sycorax Nov 29 '18 at 1:28
  • $\begingroup$ @Sycorax I agree, but what about loss and accuracy always plateauing at the same level accross different models? $\endgroup$ – hirschme Nov 29 '18 at 1:30
  • $\begingroup$ I don't think there's anything especially significant to that. It's easy to make a model that's arbitrarily bad; since it's always trying to minimize loss on the training data, I think the validation loss can only increase so much, hence the plateau. $\endgroup$ – Sycorax Nov 29 '18 at 1:44

I think more the validation loss diverging at 500 epochs in the plot you have is more noticeable than the validation accuracy plateauing. I would recommend reducing the number of hidden units and see if it changes anything, in case you have not tried this already. I would set my first objective to reach similar loss and accuracy on train and validation and then try to improve both together. If none of these helped, I would check the train/validation split. Are you shuffling the data? Another reason for the performance above could be different distribution of training and validation sets.

Edit (based on new results):

I think this is a better start now. Usually the dropout values I have seen are .2-.5. 0.95 seems to much. If a lower dropout overfits, reduce the hidden units to 100.

Here is what I might approach this. Fix the # of epochs to maybe 100, then reduce the hidden units so that after 100 epochs you get the same accuracy on training and validation, although this might be as low as 65%. This is what I call a good start.

From here, I'll try these maybe:

  1. Start increasing the hidden units. (We know this start overfitting from your data, so go to option 2.)
  2. Add more layers, starting with few hidden units. Maybe 10 or so at each layer.
  3. Try other forms of RNNs like GRU.
  4. Tweak the # of observations, lower values may not have enough information, higher values might be tough to run, taking more time and still not capturing the long-term dependencies.
  5. If you are dealing with time-series data, not sequences like text, try applying pre-processing techniques like spectrogram and see if that helps.
  6. Take a look at Attention layers. See if you can leverage those with your model.
  • $\begingroup$ I can't reduce the size of the model anymore, can overfit the training set using only two hidden layers.. shuffling the data is a clear candidate, but my question points to any other possible causes for this behavior to happen? (e.g. local minima that works for the training data) $\endgroup$ – hirschme Dec 7 '18 at 18:44
  • 1
    $\begingroup$ What you are providing as an exmaple, is basically the same as I have mentioned in the comments. Let's say that you get stock in a local minima in training. If this local minima is not performing well in test time, it means that training and test data sets have different distribution, otherwise, the test results would have been similar to that of training. $\endgroup$ – SaTa Dec 9 '18 at 21:23
  • 1
    $\begingroup$ Cool! Do you do one-hot encoding on you binary labels? If you labels are 0 and 1, you can just do a Dense(1)and then Activation("softmax"). Can you try this with default Adam and post new plots. Try 100, 400 for hidden units and dropout of .2 or .4. I want to get a sense of a base performance. $\endgroup$ – SaTa Dec 10 '18 at 2:52
  • 1
    $\begingroup$ Honestly, I think the chances are very slim. Unless you have very low variation in your data. Btw, you normalize the data, right? $\endgroup$ – SaTa Dec 11 '18 at 22:06
  • 1
    $\begingroup$ I do not have much experience with images, but this could be certainly sth I will try. If you are dealing with images, I highly recommend trying CNN/LSTM and ConvLSTM rather than treating each image as a giant feature vector. $\endgroup$ – SaTa Dec 11 '18 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.