How to choose between an overfit model and a non-overfit model? I often encounter this situation in modeling. Suppose I build two classification models. Below is their performance:
Model 1: training accuracy: 0.80, test accuracy: 0.50
Model 2: training accuracy: 0.53, test accuracy: 0.47
Obviously model 1 is overfitted and model 2 is not. However, model 1 outperforms model 2 in terms of test accuracy. So which one should be selected for production deployment?
 A: First of all, you need to choose before the final test. The purpose of the final test is to measure/estimate generalization error for the already chosen model.
If you choose again based on the test set, you either

*

*need to restrict yourself to not claim any generalization error. I.e. you can say that your optimization heuristic yielded model x, but you cannot give an estimate of generalization error for model x (you can only give your test set accuracy as training error since such a selection is part of training)

*or you need to get another test set that is independent of the whole training procedure including selecting between your two candidate models, and then measure generalization error for the final chosen model with this third test set.


Secondly, you need to make sure that the more overfit model actually outperforms the less overfit model in the test: Test set results do have random uncertainty, and this is known to be large for figures of merit like accuracy which are proportions of tested cases. This means that substantial numbers of tested cases are required to guide such a decision between two models based on accuracy.
In the example, a difference such an in the question can easily need several thousand test cases to be significant (depends on the actual distribution of correct/wrong predictions for both models, and on whether only those 2 models are compared).
Other figures of merit, in particular proper scoring rules, are much better suited to guide selection decisions. They also often have less random uncertainty than proportions.
If model 2 turns out not to be significantly better*, I'd recommend choosing the less complex/less overfit model 1.
Essentially this is also the heuristic behind the one-standard-deviation rule: when uncertain, choose the less complex model.

* Strictly speaking, significance only tells us the probability to observe at least such a difference iff there is really no difference in the performance [or if model 2 is really no better than model 1], while we'd like to decide based on the probability that model 2 is better than model 1 - which we cannot access without further information or assumptions about the pre-test probability of model 2 being better than model 1.
Nevertheless, accounting for this test set size uncertainty via significance is a big step into the right direction.
A: This is impossible to answer without more information. Class balance, tolerance for false positive/negative results, etc are important factors into deciding if model is for for production.
I've seen models with a very high accuracy score poorly on something like MCC due to most of the predictions being wrong on the minority class, which in our case was the most important class to get right.
In any case look at the confusion matrix and ask yourself how is each model doing relative to your specific use case and tolerance for error. Maybe that will give you a better intuition.
A: Overfit or not, you should pick the one with the highest test accuracy, conditional on the fact that you have truly kept your test data separate. I would be tempted to find more unseen test data to double check that it has truly generalised well to new data.
