3
$\begingroup$

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.

But how to distinguish whether a model is overfitting or underfitting from the AUC curve or AUC value of Training, test and desired AUC values?

$\endgroup$

1 Answer 1

7
$\begingroup$

Alone, the ROC curve (or AUC) of the final model on the training set will not provide any information (unless you know something about the performance of the optimal classifier). By definition, the training set cannot be used to evaluate overfitting/underfitting, as it cannot measure the generalization performance of the model. However, comparing the ROC curves of the training set and the validation set can help. The size of the gap between the training and validation metrics is an indicator of overfitting when the gap is large, and indicates underfitting when there is no gap. Everything in between is subject to interpretation, but a good model should produce a small gap.

Measuring the gap between the training and validation ROC curves should be done by measuring the area between the curves. Keep in mind that the difference between the AUCs does not compute the same quantities.

Monitoring the ROC curves (and the gap) during the learning phase can bring additional information as you can see the gap size progression.

$\endgroup$
3
  • $\begingroup$ Hi @user3421119, thanks for your answer but im not sure what you meant by "Keep in mind that the difference between the AUCs does not compute the same quantities." If you can help me with that? $\endgroup$ Jan 30, 2019 at 10:37
  • 1
    $\begingroup$ Two ROC curves can have the same AUC without being identical. The area between the two curves can be positive while the AUC difference is 0. Both can be used, but they are not the same. $\endgroup$
    – cortax
    Jan 30, 2019 at 14:53
  • 1
    $\begingroup$ Thanks @cortax,thats much helpful. I got the point. Here is my conclusion: If Training AUC value and Test/validation AUC value differs significantly then it is probably Overfitting and in case they do not differ significantly but have pretty low values (say around 0.5-0.6) then it is probably Underfitting. Let me know if i got it right. $\endgroup$ Jan 31, 2019 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.