Relation between AUROC and threshold

Question

As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that for 70% of all new data points positive class will be scored higher, than negative class (am I wrong in this last sentence?).

I'm interested in the particular situation when you have train and test data, and for both you have AUROC = 1, BUT for train dataset the threshold that perfectly separates two classes is 0.5, and for test dataset the threshold that perfectly separates two classes is 0.18. Completely different thresholds, but the same AUROCs. Is this even possible?

My motivation for this question is that I can't see why to check AUROC, if the thresholds that perfectly separates the classes for different data may be different (may be these are two completely different topics and I confuse them?). I mean, if we know AUROC = 1, then we expect our binary classifier to always perfectly separate two classes. But from the single value of AUROC we can't say what threshold to choose so that our classes are perfectly separated. So, AUROC = 1 tells us that our classifier IS ABLE to perfectly separate the data, but we also have to learn what threshold produces this separation. And indeed suppose our classifier always perfectly separates the two classes for any given data, but the threshold producing this separation is always different (first time it's 0.5, next time it's 0.18 and so on). Then how does the value of AUROC = 1 help us, if the threshold can change for different data? The data will always be perfectly separated, but we don't know by which threshold.

My guess is that AUROC estimates expected proportion of new data for which positive classes will be scored higher, than negative classes, and threshold can't change that wildly because we expect the data we use during the training to be representative. But I'm not sure.

I examined this site, but wasn't able to find an answer to this particular question. Can you answer it or just show me the direction where I'm wrong?

---

EXTRA: some pictures, illustrating my question.

Train dataset with threshold = 0.5:

Test dataset with threshold = 0.18:

Answer 1

It will be more enlightening to use the definition of the AUROC than a derivative property like the probabilities of positive/negative classes at certain threshold levels.

Recall that the AUROC is defined as double the area under the precision/recall curve as the threshold varies over all possible values. Thus, the AUROC has nothing to do with specific threshold values, since we integrate the threshold out when calculating the AUROC by straightforward integration.

Next, recall how the precision-recall curve looks like if there is a threshold that perfectly separates your data: it goes from (0,0) to (0,1) and then to (1,1). The specific value of the separating threshold only enters as the point where the parameterized curve jumps to (0,1).

Thus, if you have two different models (or datasets), each of which has a separating threshold, the precision-recall curves are identical. And therefore, so is the AUROC (and it is always 1).

Finally, therefore the AUROC is no help in determining an "optimal" threshold. There are various ways of doing so based on the original precision-recall curve (essentially as a tradeoff between precision and recall), but these also break down in degenerate cases - and in any case I would argue that any determination of thresholds that is solely based on the statistics is mistaken.