1
$\begingroup$

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.

But how to distinguish whether a model is overfitting or underfitting from the AUC curve or AUC value of Training, test and desired AUC values?

$\endgroup$
4
$\begingroup$

Alone, the ROC curve (or AUC) of the final model on the training set will not provide any information (unless you know something about the performance of the optimal classifier). By definition, the training set cannot be used to evaluate overfitting/underfitting, as it cannot measure the generalization performance of the model. However, comparing the ROC curves of the training set and the validation set can help. The size of the gap between the training and validation metrics is an indicator of overfitting when the gap is large, and indicates underfitting when there is no gap. Everything in between is subject to interpretation, but a good model should produce a small gap.

Measuring the gap between the training and validation ROC curves should be done by measuring the area between the curves. Keep in mind that the difference between the AUCs does not compute the same quantities.

Monitoring the ROC curves (and the gap) during the learning phase can bring additional information as you can see the gap size progression.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Hi @user3421119, thanks for your answer but im not sure what you meant by "Keep in mind that the difference between the AUCs does not compute the same quantities." If you can help me with that? $\endgroup$ – heisenbug47 Jan 30 '19 at 10:37
  • $\begingroup$ Two ROC curves can have the same AUC without being identical. The area between the two curves can be positive while the AUC difference is 0. Both can be used, but they are not the same. $\endgroup$ – cortax Jan 30 '19 at 14:53
  • $\begingroup$ Thanks @cortax,thats much helpful. I got the point. Here is my conclusion: If Training AUC value and Test/validation AUC value differs significantly then it is probably Overfitting and in case they do not differ significantly but have pretty low values (say around 0.5-0.6) then it is probably Underfitting. Let me know if i got it right. $\endgroup$ – heisenbug47 Jan 31 '19 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.