Can I still use an overfitted model with high test accuracy?

Question

Below is the training statistics output from training a Keras/TF model. You can see val_accuracy peaks at Epoch 4 with 0.6633. After that accuracy(train) continues to go up but val_accuracy becomes worse/lower. I generated two models: model_4 from 4 epoch runs, model_10 from 10 epoch runs. Then the test dataset is applied to them to compute test_accuracy. I got 0.7040 for model_4 and 0.7152 for model_10. Apparently from the training statistics, model_10 is overfitting the training set. However its test_accuracy doesn't deteriorate. Can I still choose model_10 for the production deployment? What is the implication of using an overfitted model?

Epoch 1/10
# 703/703 [==============================] - 821s 1s/step - loss: 1.0799 - accuracy: 0.6451 - val_loss: 1.0686 - val_accuracy: 0.6306
# Epoch 2/10
# 703/703 [==============================] - 821s 1s/step - loss: 0.8158 - accuracy: 0.7277 - val_loss: 0.9982 - val_accuracy: 0.6488
# Epoch 3/10
# 703/703 [==============================] - 820s 1s/step - loss: 0.6541 - accuracy: 0.7798 - val_loss: 0.9794 - val_accuracy: 0.6616
Epoch 4/10
703/703 [==============================] - 821s 1s/step - loss: 0.5162 - accuracy: 0.8340 - val_loss: 0.9962 - val_accuracy: 0.6633
Epoch 5/10
703/703 [==============================] - 821s 1s/step - loss: 0.3926 - accuracy: 0.8840 - val_loss: 1.0285 - val_accuracy: 0.6587
Epoch 6/10
703/703 [==============================] - 821s 1s/step - loss: 0.2804 - accuracy: 0.9298 - val_loss: 1.0956 - val_accuracy: 0.6616
Epoch 7/10
703/703 [==============================] - 821s 1s/step - loss: 0.1864 - accuracy: 0.9629 - val_loss: 1.1609 - val_accuracy: 0.6584
Epoch 8/10
703/703 [==============================] - 820s 1s/step - loss: 0.1155 - accuracy: 0.9825 - val_loss: 1.2688 - val_accuracy: 0.6502
Epoch 9/10
703/703 [==============================] - 820s 1s/step - loss: 0.0698 - accuracy: 0.9923 - val_loss: 1.3471 - val_accuracy: 0.6493
Epoch 10/10
703/703 [==============================] - 820s 1s/step - loss: 0.0445 - accuracy: 0.9952 - val_loss: 1.4566 - val_accuracy: 0.6488

```

Answer 1

That means your test set mimics the training set better than your validation set. Considering the size of your training dataset (i.e. 703) this is possible. And, that's not guaranteed in production. What if you were using your test set as validation, and your validation set as test? In that case, the situation would be the opposite, and you'd have chosen the model with 4 epochs (ignoring the other epochs for the sake of simplicity).

The implication is that overfitting means high variance. So, in production, if your model is tested with a lot of samples, model 10 will have more variance in its decisions than model 4. Considering the gap between training and validation performances, both cases have some degree of overfitting, but situation in model 10 seems to be more serious.

Apart from choosing amongst these two, you may also use cross-validation to either select the best epoch, or generate $k$ different models and average/vote their predictions. Considering the size of your data, this approach seems possible and will increase the stability of your predictions.