Return to Question

In practice, there can be a classifier that gives far better performance at a specific acceptable threshold than an "optimal" classifier with better average performance across range of thresholds (higher AUC) but not so much at that threshold.

For example: Classifier 1: 80% TPR with 5% FPR; 95% TPR with 40%FPR AUC : .6 Classifier 2: 40% TPR with 5% FPR; 95% TPR with 20%FPR AUC : .9

Shouldn't i use classifier 1 instead of classifier 2 if i am operating around 5% FPR acceptable threshold?

Also, what should if allowed to run near 10% FPR?

• ( a ) Should i just check TPR from both classifiers corresponding to 10% FPR and pick the classifier that has higher TPR?
• ( b ) Compute a partial area under curve till 10% FPR and pick one with highest

In practice, there can be a classifier that gives far better performance at a specific acceptable threshold than an "optimal" classifier with better average performance across range of thresholds (higher AUC) but not so much at that threshold.

For example:

• Classifier 1: 80% TPR with 5% FPR, 95% TPR with 40% FPR, AUC = 0.6
• Classifier 2: 40% TPR with 5% FPR, 95% TPR with 20% FPR, AUC = 0.9

Shouldn't I use classifier 1 instead of classifier 2 if I am operating around 5% FPR acceptable threshold?

Also, what if I am allowed to run near 10% FPR?

• Should I just check TPR from both classifiers corresponding to 10% FPR and pick the classifier that has higher TPR?
• Or compute a partial area under curve until 10% FPR and pick the one with the highest?