Is overfitted model with higher AUC on test sample better than not overfitted one

Question

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set.
The other participant has created a model that performs 96% AUC on train set and 76% AUC on hold-out test set.

In my opinion my model is better because it performs on the same level on a hold out set.
Q: Is it valid to argue that his 76% on a test set is just a coincidence and on another hold out test set his model could perform worse?

Answer 1

This will depend on how your training and test sets are composed.

If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).

Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the model has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.

And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.

Answer 2

If the focus is purely on predictive accuracy, then the overfitted model is most probably better. Take e.g. a random forest: On the training data set, by construction, it extremely overfits. Still the results on the test data set are often quite reasonable (and the test performance close to the stated out-of-bag performance).

This works only if the test data set reflects "real cases" and the assumptions of the underlying models are met reasonably.

Answer 3

It's quite possible (and in certain situations) to be overfitting on the test set as well. Properly fit models should achieve approximately similar cross validated performance on both the training and test datasets. Best practices would be to also hold out another portion of the dataset that is only used once: to assess the performance of the model on data it hasn't seen at all.

If you're using the test set to build the model iteratively, say adding a feature then seeing how it validates on the test set, you're giving the model information about the test set. Specifically you're biasing your results on the test set to be higher (that is, you're overfitting) if you tune the model based on its test set performance.