# How to choose between an overfit model and a non-overfit model?

I often encounter this situation in modeling. Suppose I build two classification models. Below is their performance:

Model 1: training accuracy: 0.80, test accuracy: 0.50
Model 2: training accuracy: 0.53, test accuracy: 0.47

Obviously model 1 is overfitted and model 2 is not. However, model 1 outperforms model 2 in terms of test accuracy. So which one should be selected for production deployment?

• Nov 1 '20 at 1:47

First of all, you need to choose before the final test. The purpose of the final test is to measure/estimate generalization error for the already chosen model.

If you choose again based on the test set, you either

• need to restrict yourself to not claim any generalization error. I.e. you can say that your optimization heuristic yielded model x, but you cannot give an estimate of generalization error for model x (you can only give your test set accuracy as training error since such a selection is part of training)
• or you need to get another test set that is independent of the whole training procedure including selecting between your two candidate models, and then measure generalization error for the final chosen model with this third test set.

Secondly, you need to make sure that the more overfit model actually outperforms the less overfit model in the test: Test set results do have random uncertainty, and this is known to be large for figures of merit like accuracy which are proportions of tested cases. This means that substantial numbers of tested cases are required to guide such a decision between two models based on accuracy.

In the example, a difference such an in the question can easily need several thousand test cases to be significant (depends on the actual distribution of correct/wrong predictions for both models, and on whether only those 2 models are compared).

Other figures of merit, in particular proper scoring rules, are much better suited to guide selection decisions. They also often have less random uncertainty than proportions.

If model 2 turns out not to be significantly better*, I'd recommend choosing the less complex/less overfit model 1.

Essentially this is also the heuristic behind the one-standard-deviation rule: when uncertain, choose the less complex model.

* Strictly speaking, significance only tells us the probability to observe at least such a difference iff there is really no difference in the performance [or if model 2 is really no better than model 1], while we'd like to decide based on the probability that model 2 is better than model 1 - which we cannot access without further information or assumptions about the pre-test probability of model 2 being better than model 1.
Nevertheless, accounting for this test set size uncertainty via significance is a big step into the right direction.

• I'd also be included to call this a train and validation set, since you're making a decision which model to select on the basis of the second set. Then as you mention you estimate the final accuracy of the selected model using the test set. Some frameworks (i.e. pytorch lightning use the train/validatiion/test terminology) but others (i.e. skleanr) tend to talk about train / test despite often the use of the test set is for hyperparameter and/or model selection in which case validation is probably more correct terminology. Nov 1 '20 at 5:18
• @DavidWaterworth: I beg to disagree in the sense that IMHO the term validation set for a particular training set is highly misleading (and some other term such as test set for data sets for validation purposes). In my courses I avoid this terminology (other than explaining that it exists and is confusing): I use only training vs. verification (to avoid yet more confusion with validation), and introduce a high-level training concept that encapsulates whatever is necessary to train the one final production model. I.e. that training procedure may internally split data into "fitting" and... Nov 1 '20 at 18:47
• ... "optimization" sets. But the verification (model meets performance criteria) and validation (in the original sense of making sure the model is fit for its purpose) looks at model training as a black box and thus does not need anything but the data needed for verification (validation takes care that the right data is used for verification). Nov 1 '20 at 18:52

This is impossible to answer without more information. Class balance, tolerance for false positive/negative results, etc are important factors into deciding if model is for for production.

I've seen models with a very high accuracy score poorly on something like MCC due to most of the predictions being wrong on the minority class, which in our case was the most important class to get right.

In any case look at the confusion matrix and ask yourself how is each model doing relative to your specific use case and tolerance for error. Maybe that will give you a better intuition.

Overfit or not, you should pick the one with the highest test accuracy, conditional on the fact that you have truly kept your test data separate. I would be tempted to find more unseen test data to double check that it has truly generalised well to new data.

• No, that implies that improper accuracy scores are relevant here. They are not. What is all important is to produce a smooth no-binning calibration curve and show the the absolute accuracy of predicted risks over the whole spectrum of predictions is acceptable. Oct 31 '20 at 13:14