I have 30k sequences of 8 letters that needs to be classified in X or Y depending on the relative position of letters in the sequence.

The features are converted to numbers via a dict mapping and duplicates are removed.

I trained the following LSTM model

lstmbi = Sequential()
lstmbi.add(Embedding(22, 128, input_length=max_length))
lstmbi.add(Bidirectional(LSTM(6, kernel_regularizer=regularizers.l2(0.00001), 
lstmbi.add(Dense(8, activation='relu'))
lstmbi.add(Dense(1, activation='sigmoid'))
opt = tf.keras.optimizers.Adam(learning_rate=0.003)
lstmbi.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])

checkpoint_name = '/content/drive/My Drive/protease_RNN/tmp_weights.hdf5'
checkpoint = ModelCheckpoint(checkpoint_name, monitor='val_accuracy', verbose = 1, save_best_only = True, mode ='max', patience=10)
callbacks_list = [checkpoint]

history= lstmbi.fit(X_train, y_train, 
                    validation_data=(X_val, y_val), 

And I monitored losses and accuracy per epoch for validation to check for over/under fitting

enter image description here

After I calculated performance on the test set

Accuracy: 0.871053
Precision: 0.802035
Recall: 0.674184
F1 score: 0.732573
ROC AUC: 0.896444

and everything looks kind of okay. But then I used this network to predict a set of 100k+ sequences where there are 145 that are ground positive (not used for training) and the rest is unlabelled and only 13 of this 145 are recovered.

I checked via PCA if this dataset overlaps with the training data to see if feature distribution is the same and it does.

What can be the source of the discrepancy between the two performances?

  • $\begingroup$ Seems like the definition of overfitting: the model does well on the training data, but not as well on new data. stats.stackexchange.com/questions/tagged/overfitting+definition Is there a reason you don't think that this is overfitting? $\endgroup$ – Sycorax Sep 12 at 23:26
  • $\begingroup$ There is independent test (X_test, y_test) which the model has never seen and performance is okay there (overfitting would be bad validation/test performance and good training performance). Seems a generalization issue but I'm not so expert in RNN models $\endgroup$ – D.A. Sep 13 at 0:29
  • $\begingroup$ A problem with how overfitting is framed in classrooms is that students leave with the impression that overfitting only ever happens if the out-of-sample performance is terrible. Cleary that's undesirable, but by anchoring the overfitting discussion in terms of the worst-case scenario, it skips over the observation that overfitting can be a matter of degrees. After all, how likely is it that you'll get training and testing data to have exactly the same statistics? Unless these statistics are exactly the same, one will be larger... $\endgroup$ – Sycorax Sep 13 at 0:35
  • $\begingroup$ Yeah might be a bit too dogmatic in that, I double checked data distribution to make sure that I'm not comparing apple (in data used for test/training/val) and oranges (in new data) and the PCA does not show any separation or cluster between this two datasets. Not so sure if I should just add more regularization and dropout $\endgroup$ – D.A. Sep 13 at 0:45
  • $\begingroup$ The purpose of test and validation data is to simulate what happens when the model encounters new data, so the distinction you're drawing is spurious. The overall distributions might be the same, but that's unrelated to whether the model has overfit to the training data. If increasing regularization (dropout, L1/L2 penalties) shrinks the gap between training and test sets, then this is consistent with overfitting as an explanation for the discrepancy. $\endgroup$ – Sycorax Sep 13 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.