2
$\begingroup$

There is one thing which confuses me about two very common explanations regarding the interpretation of the Area Under The Receiver Operating Characteristic (referred to shortly as AUC). Concretely, these are

1) The benchmark of 0.5 for a coin toss, i.e. that if a simple coin toss is determined as a "test", that in this case the AUC-value would be 0.5.

2) That the AUC-value corresponds to the probability of correctly classifying two randomly individuals, one from each group (e.g. a person who has the disease a person who does not have the disease). For the matter of this post, the alternative explanation that, using normalized units, the AUC area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative').

What confuses me is the following thought: Suppose I use the coin-toss "test" and apply it two the two individuals from property 2.). This test has a probability of 0.5 for a false negative and 0.5 for a false positive. The probability that I classify a positive and a negative outcome correctly is thus 0.25 (Prob[no false negative for the positive indivudual] * Pr[no false positive for the negative individual]). In all other cases, the two scores/tests will be either equal or have results opposite to what is actually true (interpreting the test inversely as often suggested when the area is <0.5 does not help either for the coin flip case). This seems to be at odds with statement 1).

Of course, I could just pick one of the two individuals, test it, and assign it the outcome the test shows - which is correct in 50% of all cases - and assign the other individuals the opposite outcome. This would achieve an accuracy of 0.5 as described in 1) but it would not really be what is described in 2) since I am only testing one person (i.e., one person in the sample would not have a test-score).

The two paragraphs of reasoning seem to contradict each other. Is any of them wrong, if yes where or is this really a paradox?

*Note that I see that the coin toss would be under the diagonal and would get, together with other tests where the false-positive rate equals the true-positive rate, get an area of 0.5 under curve. It is just about the interpretation aides listed above and how to reconcile them.

$\endgroup$
  • 1
    $\begingroup$ I don't see an explicit question here. In fact, the only "?" I see is "Any thoughts?" towards the bottom, which is not an actual question grammatically. Moreover, 'do you have any thoughts about this?' (which is clearly implied), is off topic within the SE system--it is too broad*. Can you clarify your question, & make it sufficiently concrete to be answerable in a couple paragraphs? $\endgroup$ – gung Apr 9 '18 at 14:09
  • $\begingroup$ I think it's just a confusion to say "the benchmark auc of 0.5 corresponds to a coin toss". It'd be better if you just called it "the benchmark auc of 0.5 corresponds to a random classifier (or something)". The coin toss examples brings in too many tie cases which are not representative in the [0, 1] probability continuum. $\endgroup$ – Stergios Apr 10 '18 at 10:59
  • $\begingroup$ it would be useful to provide a link to your source for the two statements 1) and 2) as both are imprecise (as Carl handles in his answer). Seeing how the confusion has arisen would help ensure that the answerer can address the specific underlying causes of the confusion. $\endgroup$ – ReneBt Apr 11 '18 at 8:25
2
$\begingroup$

"This test has a probability of 0.5 for a false negative and 0.5 for a false positive." No it does not; that would add up to 100% false, which would be an AUC of zero.

This paragraph is so misleading that it likely has provided a reason for closing the question.

"2) That the AUC-value corresponds to the probability of correctly classifying two randomly individuals, one from each group (e.g. a person who has the disease a person who does not have the disease). For the matter of this post, the explanation that the value corresponds to the probability of assigning a higher test score to an ex-ante determined group, say the one with a positive outcome than to other, is identical in this regard."

This should read "When using normalized units, the area under the curve (often referred to as simply the AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative')."

 EDIT following OP edit

I think I know what the conceptual problem is. False positives ($FP$) are not the false positive rate. The false positive rate is ${\frac {FP}{N}}={\frac {FP}{FP+TN}}$. Let me speak here of outcomes. The false positive fraction of outcomes is $\frac{FP}{total\; outcomes}$. For an unbiased coin, when we guess all heads and 50% of the total outcomes are false positives, that is a 100% false positive rate. The inversion of 0.5 for a false negative fraction and 0.5 for a false positive fraction is 50% true positives and 50% true negatives, which, counting heads, and unless you have trouble adjudicating a coin toss while you are looking straight at it, is what you would expect for absolute certainty for 100% of the tosses. What the $x=y$ line represents is, for example, being blindfolded and guessing what the coin toss outcomes are. So if half the time, you guess heads and half the time you guess tails, and you are right half the time in which case $\frac{1}{4}$ are true positives, $\frac{1}{4}$ are true negatives, $\frac{1}{4}$ are false positives, and $\frac{1}{4}$ are false negatives. Now if you jigger this and guess more heads than tails, your false positives increase, e.g. to $\frac{1}{3}$, false negatives decrease, e.g., to $\frac{1}{6}$, true positives increase and true negatives decrease, but you are still only right at $\frac{1}{2}$ true or false, i.e., $\frac{1}{3}+\frac{1}{6}$. In this way, if you guessed heads all the time, you would have no false negatives and 50% false positives. So, by varying the percentage of guesses that are heads you could plot out the entire ROC $y=x$ curve, with an AUC of 50%, and THAT is what it means.

Note that an AUC of 50% is an exactly useless test, and irredeemably so. One would be better off misunderstanding a test to obtain an AUC of $\frac{1}{3}$, because then, as here, all one has to do is flip the understanding $180^{\circ}$ to obtain an AUC of $\frac{2}{3}$, i.e., useful knowledge.

$\endgroup$
  • $\begingroup$ Thanks for your comments Carl. I incorporarted some of the suggested changes. I think that the 50% false negative plus 50% are accurate description of a coin toss and do not app to 100% false since they are both conditional on having to test true a positive or negative outcome, respectively. So I left it. (Also, even if it were true and would result in an AUC of 0 for a coin toss, it's interpretation could be inverted and it were perfect - which is clearly not the case). $\endgroup$ – Han Apr 10 '18 at 10:08
  • $\begingroup$ @Han isn't the inversion 50% true positives and 50% true negatives, which, counting heads, and unless you have trouble adjudicating a coin toss while you are looking straight at it, is what you would expect for absolute certainty for 100% of the tosses. $\endgroup$ – Carl Apr 10 '18 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.