I recently learned AUC and ROC and confused about the usage of AUC. What if my model A has a higher AUC than my model B but in terms of one particular threshold--one particular point on the ROC graph--model B has the highest accuracy or F1 score? If this situation happens, then although the overall performance of model A is better, I will choose model B because my prediction will be based on one model with a particular threshold.
Is my logic right? Will the situation I described happen? Thanks
AUC has a literal interpretation: it is the chance that a randomly chosen negative sample ranks lower than a randomly chosen positive sample. It is a heuristic diagnostic for your model's performance. As you stated, AUC doesn't imply you have a better choice of threshold (which is an operational decision based on the tradeoff you need). So you can easily have a model with higher AUC but worse threshold values for your purposes. For example, suppose you can only afford a 20% FP rate. Two models might have very similar threshold performance at the 20% FP rate, but one model might have higher AUC due to better performance past the 20% threshold. In your case this is worthless.
