# Relation between AUROC and threshold

As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that for 70% of all new data points positive class will be scored higher, than negative class (am I wrong in this last sentence?).

I'm interested in the particular situation when you have train and test data, and for both you have AUROC = 1, BUT for train dataset the threshold that perfectly separates two classes is 0.5, and for test dataset the threshold that perfectly separates two classes is 0.18. Completely different thresholds, but the same AUROCs. Is this even possible?

My motivation for this question is that I can't see why to check AUROC, if the thresholds that perfectly separates the classes for different data may be different (may be these are two completely different topics and I confuse them?). I mean, if we know AUROC = 1, then we expect our binary classifier to always perfectly separate two classes. But from the single value of AUROC we can't say what threshold to choose so that our classes are perfectly separated. So, AUROC = 1 tells us that our classifier IS ABLE to perfectly separate the data, but we also have to learn what threshold produces this separation. And indeed suppose our classifier always perfectly separates the two classes for any given data, but the threshold producing this separation is always different (first time it's 0.5, next time it's 0.18 and so on). Then how does the value of AUROC = 1 help us, if the threshold can change for different data? The data will always be perfectly separated, but we don't know by which threshold.

My guess is that AUROC estimates expected proportion of new data for which positive classes will be scored higher, than negative classes, and threshold can't change that wildly because we expect the data we use during the training to be representative. But I'm not sure.

I examined this site, but wasn't able to find an answer to this particular question. Can you answer it or just show me the direction where I'm wrong?

### EXTRA: some pictures, illustrating my question.

Train dataset with threshold = 0.5: Test dataset with threshold = 0.18: • (1) How do you determine the "best" threshold? (2) Be aware that determining thresholds should include the costs of decisions. (3) I don't quite understand what you mean by "I can't see why to check AUROC, if the best thresholds for different data may be different", can you clarify? What does it mean that AUC is a semi-proper scoring rule? may be helpful. Nov 23 at 10:25
• @StephanKolassa, Hello and thank you for the comment! I edited my question: simplified the example (it still represents the same issue). (1) I removed "best" from the question, since it can be misleading. Initially by "best" I intended to mean "threshold that perfectly separates two classes", but my example was not good. (3) I wrote a clarification in the same paragraph. Also changed the pictures to simplify my question. Nov 23 at 10:59
• @StephanKolassa, Actually my confusion is that for both datasets in the picture two classes are perfectly separated, but if we use the first threshold (0.5) for the second dataset we can't expect the perfect separation, since all the data points will end up under the threshold = 0.5. Nov 23 at 11:04
• @StephanKolassa, I mean, AUROC does not tell us the proportion of the data, that will be classified correctly, since "correct" depends on the chosen threshold. AUROC only tell us the expected proportion of the data for which positive classes will be scored higher, than negative ones. But why is this knowledge even meaningful, if for different data the two classes may turn out to be classified correctly but by the different threshold? Hope, all this clarifies my question. If not, let me know, please. Nov 23 at 11:15