8
$\begingroup$

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set.
The other participant has created a model that performs 96% AUC on train set and 76% AUC on hold-out test set.

In my opinion my model is better because it performs on the same level on a hold out set.
Q: Is it valid to argue that his 76% on a test set is just a coincidence and on another hold out test set his model could perform worse?

$\endgroup$
2
  • 1
    $\begingroup$ How would you argue only his result could be achieved by chance, not yours? $\endgroup$
    – Firebug
    Jun 27 '16 at 17:48
  • $\begingroup$ I had the same performance in trainset, testset and hold-out set. He had 96% in trainset, I don't know what was his performance in testset and he had 76% in hold-out set. So for me it looks like my model is producing stable results while his is overfitted to trainset and I am not sure if given new sample his model would produce same 76%. $\endgroup$
    – MiksL
    Jun 28 '16 at 7:29
7
$\begingroup$

This will depend on how your training and test sets are composed.

If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).

Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the model has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.

And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.

$\endgroup$
4
$\begingroup$

If the focus is purely on predictive accuracy, then the overfitted model is most probably better. Take e.g. a random forest: On the training data set, by construction, it extremely overfits. Still the results on the test data set are often quite reasonable (and the test performance close to the stated out-of-bag performance).

This works only if the test data set reflects "real cases" and the assumptions of the underlying models are met reasonably.

$\endgroup$
1
  • $\begingroup$ The focus is to understand which model would perform better in future if put into a production mode. $\endgroup$
    – MiksL
    Jun 27 '16 at 10:32
0
$\begingroup$

It's quite possible (and in certain situations) to be overfitting on the test set as well. Properly fit models should achieve approximately similar cross validated performance on both the training and test datasets. Best practices would be to also hold out another portion of the dataset that is only used once: to assess the performance of the model on data it hasn't seen at all.

If you're using the test set to build the model iteratively, say adding a feature then seeing how it validates on the test set, you're giving the model information about the test set. Specifically you're biasing your results on the test set to be higher (that is, you're overfitting) if you tune the model based on its test set performance.

$\endgroup$
1
  • 3
    $\begingroup$ (-1) Sorry, I don't agree with this "Properly fit models should achieve approximately similar cross validated performance on both the training and test datasets". Random Forests routinely achieve perfect scores on train data, for example, are you saying they are not properly fit? $\endgroup$
    – Firebug
    Jun 27 '16 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.