I have 30k sequences of 8 letters that needs to be classified in X or Y depending on the relative position of letters in the sequence.
The features are converted to numbers via a dict mapping and duplicates are removed.
I trained the following LSTM model
lstmbi = Sequential() lstmbi.add(Input(shape=(8,))) lstmbi.add(Embedding(22, 128, input_length=max_length)) lstmbi.add(Bidirectional(LSTM(6, kernel_regularizer=regularizers.l2(0.00001), recurrent_regularizer=regularizers.l2(0.00001), bias_regularizer=regularizers.l2(0.00001)))) lstmbi.add(Dropout(0.3)) lstmbi.add(Dense(8, activation='relu')) lstmbi.add(Dropout(0.25)) lstmbi.add(Dense(1, activation='sigmoid')) opt = tf.keras.optimizers.Adam(learning_rate=0.003) lstmbi.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy']) checkpoint_name = '/content/drive/My Drive/protease_RNN/tmp_weights.hdf5' checkpoint = ModelCheckpoint(checkpoint_name, monitor='val_accuracy', verbose = 1, save_best_only = True, mode ='max', patience=10) callbacks_list = [checkpoint] history= lstmbi.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100, batch_size=128, callbacks=callbacks_list, verbose=1)
And I monitored losses and accuracy per epoch for validation to check for over/under fitting
After I calculated performance on the test set
Accuracy: 0.871053 Precision: 0.802035 Recall: 0.674184 F1 score: 0.732573 ROC AUC: 0.896444
and everything looks kind of okay. But then I used this network to predict a set of 100k+ sequences where there are 145 that are ground positive (not used for training) and the rest is unlabelled and only 13 of this 145 are recovered.
I checked via PCA if this dataset overlaps with the training data to see if feature distribution is the same and it does.
What can be the source of the discrepancy between the two performances?