I recently learned AUC and ROC and confused about the usage of AUC. What if my model A has a higher AUC than my model B but in terms of one particular threshold--one particular point on the ROC graph--model B has the highest accuracy or F1 score? If this situation happens, then although the overall performance of model A is better, I will choose model B because my prediction will be based on one model with a particular threshold.

Is my logic right? Will the situation I described happen? Thanks


AUC has a literal interpretation: it is the chance that a randomly chosen negative sample ranks lower than a randomly chosen positive sample. It is a heuristic diagnostic for your model's performance. As you stated, AUC doesn't imply you have a better choice of threshold (which is an operational decision based on the tradeoff you need). So you can easily have a model with higher AUC but worse threshold values for your purposes. For example, suppose you can only afford a 20% FP rate. Two models might have very similar threshold performance at the 20% FP rate, but one model might have higher AUC due to better performance past the 20% threshold. In your case this is worthless.

| cite | improve this answer | |
  • $\begingroup$ I am still confused. So if I am going to compute AUC and F1 score for my model. AUC is used to compare different models. And F1 score is used to choose the best threshold for my chosen model. Sounds like I should do AUC first and then calculate the F1 score? Is this correct? $\endgroup$ – Evan Liu Apr 24 '17 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.