1
$\begingroup$

As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that for 70% of all new data points positive class will be scored higher, than negative class (am I wrong in this last sentence?).

I'm interested in the particular situation when you have train and test data, and for both you have AUROC = 1, BUT for train dataset the threshold that perfectly separates two classes is 0.5, and for test dataset the threshold that perfectly separates two classes is 0.18. Completely different thresholds, but the same AUROCs. Is this even possible?

My motivation for this question is that I can't see why to check AUROC, if the thresholds that perfectly separates the classes for different data may be different (may be these are two completely different topics and I confuse them?). I mean, if we know AUROC = 1, then we expect our binary classifier to always perfectly separate two classes. But from the single value of AUROC we can't say what threshold to choose so that our classes are perfectly separated. So, AUROC = 1 tells us that our classifier IS ABLE to perfectly separate the data, but we also have to learn what threshold produces this separation. And indeed suppose our classifier always perfectly separates the two classes for any given data, but the threshold producing this separation is always different (first time it's 0.5, next time it's 0.18 and so on). Then how does the value of AUROC = 1 help us, if the threshold can change for different data? The data will always be perfectly separated, but we don't know by which threshold.

My guess is that AUROC estimates expected proportion of new data for which positive classes will be scored higher, than negative classes, and threshold can't change that wildly because we expect the data we use during the training to be representative. But I'm not sure.

I examined this site, but wasn't able to find an answer to this particular question. Can you answer it or just show me the direction where I'm wrong?


EXTRA: some pictures, illustrating my question.

Train dataset with threshold = 0.5:

enter image description here

Test dataset with threshold = 0.18:

enter image description here

$\endgroup$
4
  • $\begingroup$ (1) How do you determine the "best" threshold? (2) Be aware that determining thresholds should include the costs of decisions. (3) I don't quite understand what you mean by "I can't see why to check AUROC, if the best thresholds for different data may be different", can you clarify? What does it mean that AUC is a semi-proper scoring rule? may be helpful. $\endgroup$ Nov 23 at 10:25
  • $\begingroup$ @StephanKolassa, Hello and thank you for the comment! I edited my question: simplified the example (it still represents the same issue). (1) I removed "best" from the question, since it can be misleading. Initially by "best" I intended to mean "threshold that perfectly separates two classes", but my example was not good. (3) I wrote a clarification in the same paragraph. Also changed the pictures to simplify my question. $\endgroup$
    – mathgeek
    Nov 23 at 10:59
  • $\begingroup$ @StephanKolassa, Actually my confusion is that for both datasets in the picture two classes are perfectly separated, but if we use the first threshold (0.5) for the second dataset we can't expect the perfect separation, since all the data points will end up under the threshold = 0.5. $\endgroup$
    – mathgeek
    Nov 23 at 11:04
  • $\begingroup$ @StephanKolassa, I mean, AUROC does not tell us the proportion of the data, that will be classified correctly, since "correct" depends on the chosen threshold. AUROC only tell us the expected proportion of the data for which positive classes will be scored higher, than negative ones. But why is this knowledge even meaningful, if for different data the two classes may turn out to be classified correctly but by the different threshold? Hope, all this clarifies my question. If not, let me know, please. $\endgroup$
    – mathgeek
    Nov 23 at 11:15
2
$\begingroup$

It will be more enlightening to use the definition of the AUROC than a derivative property like the probabilities of positive/negative classes at certain threshold levels.

Recall that the AUROC is defined as double the area under the precision/recall curve as the threshold varies over all possible values. Thus, the AUROC has nothing to do with specific threshold values, since we integrate the threshold out when calculating the AUROC by straightforward integration.

Next, recall how the precision-recall curve looks like if there is a threshold that perfectly separates your data: it goes from (0,0) to (0,1) and then to (1,1). The specific value of the separating threshold only enters as the point where the parameterized curve jumps to (0,1).

Thus, if you have two different models (or datasets), each of which has a separating threshold, the precision-recall curves are identical. And therefore, so is the AUROC (and it is always 1).

Finally, therefore the AUROC is no help in determining an "optimal" threshold. There are various ways of doing so based on the original precision-recall curve (essentially as a tradeoff between precision and recall), but these also break down in degenerate cases - and in any case I would argue that any determination of thresholds that is solely based on the statistics is mistaken.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.