How can I better choose my model?

Question

I've made several experimentations in a classification problem, I've got these results:

ROC-AUC Metric:

1. Train: 0.99, CV: 0.95 +/- 0.01, Test: 0.96
2. Train: 0.97, CV: 0.94 +/- 0.01, Test: 0.94
3. Train: 0.93, CV: 0.93 +/- 0.00, Test: 0.93

Now that I have to choose among those which one to deploy in my app, I am wondering if I should take the 3rd because it shows no over-fitting, or take the 1st because in spite of the gap (3%) between Train & Validation the CV show a stable roc-auc and the Test confirms it. or the 2nd because it's something in between the two ?

What would you recommand ?

Answer 1

I don't think there is serious amount of overfit in any of the cases. There will always be some degree of overfitting, but the extent of it matters. In this case, you should do what you'd do for hyper-parameter tuning, i.e. choose the one with the highest validation performance. And, fortunately, your test results back up this decision. Choosing option three just because training and validation performances are equal may also justify the selection of any underfitted model.