I often encounter this situation in modeling. Suppose I build two classification models. Below is their performance:
Model 1: training accuracy: 0.80, test accuracy: 0.50
Model 2: training accuracy: 0.53, test accuracy: 0.47
Obviously model 1 is overfitted and model 2 is not. However, model 1 outperforms model 2 in terms of test accuracy. So which one should be selected for production deployment?
First of all, you need to choose before the final test. The purpose of the final test is to measure/estimate generalization error for the already chosen model.
If you choose again based on the test set, you either
Secondly, you need to make sure that the more overfit model actually outperforms the less overfit model in the test: Test set results do have random uncertainty, and this is known to be large for figures of merit like accuracy which are proportions of tested cases. This means that substantial numbers of tested cases are required to guide such a decision between two models based on accuracy.
In the example, a difference such an in the question can easily need several thousand test cases to be significant (depends on the actual distribution of correct/wrong predictions for both models, and on whether only those 2 models are compared).
Other figures of merit, in particular proper scoring rules, are much better suited to guide selection decisions. They also often have less random uncertainty than proportions.
If model 2 turns out not to be significantly better*, I'd recommend choosing the less complex/less overfit model 1.
Essentially this is also the heuristic behind the one-standard-deviation rule: when uncertain, choose the less complex model.
* Strictly speaking, significance only tells us the probability to observe at least such a difference iff there is really no difference in the performance [or if model 2 is really no better than model 1], while we'd like to decide based on the probability that model 2 is better than model 1 - which we cannot access without further information or assumptions about the pre-test probability of model 2 being better than model 1.
Nevertheless, accounting for this test set size uncertainty via significance is a big step into the right direction.
train and validation set, since you're making a decision which model to select on the basis of the second set. Then as you mention you estimate the final accuracy of the selected model using the test set. Some frameworks (i.e. pytorch lightning use the train/validatiion/test terminology) but others (i.e. skleanr) tend to talk about train / test despite often the use of the test set is for hyperparameter and/or model selection in which case validation is probably more correct terminology.
$\endgroup$
Commented
Nov 1, 2020 at 5:18
This is impossible to answer without more information. Class balance, tolerance for false positive/negative results, etc are important factors into deciding if model is for for production.
I've seen models with a very high accuracy score poorly on something like MCC due to most of the predictions being wrong on the minority class, which in our case was the most important class to get right.
In any case look at the confusion matrix and ask yourself how is each model doing relative to your specific use case and tolerance for error. Maybe that will give you a better intuition.
Overfit or not, you should pick the one with the highest test accuracy, conditional on the fact that you have truly kept your test data separate. I would be tempted to find more unseen test data to double check that it has truly generalised well to new data.
