5
$\begingroup$

Suppose I have fitted a Logistic regression model that predicts $P(Y=1|\boldsymbol{X})$ the presence of a disease which is encoded to $1$, and if not then $0$. The AUROC (area under the roc curve) shows a high discriminatory power say: $85\%$. So any randomly chosen person with the disease will have a higher predicted probability than a person without the disease - $85\%$ of the time.

If the regression model gives me a subject $A$ with a predicted probability of $0.6$ and this seems to be a high probability compared to other subjects.

Would it be correct to say that there is $85\%$ chance that $A$ has the disease?

Can you give me some examples on how I can utilize my regression model knowing that it has strong discriminatory power?

$\endgroup$
  • $\begingroup$ I'm pretty sure parts of this one were asked before, just can't find it. $\endgroup$ – Firebug Sep 22 '17 at 15:23
9
$\begingroup$

Would it be correct to say that there is 85% chance that $A$ has the disease?

No. Assuming your model is correct and well-calibrated, the probability that $A$ has the disease is the model's estimate that $A$ has the disease.

The meaning of AUROC (area under the ROC curve, to distinguish from the less-common area under the precision-recall curve) is exactly what you state: given a randomly-selected diseased person and a randomly-selected healthy person, there is an 85% chance that your model ranks the diseased person higher than the healthy person.

Can you give me some examples on how I can utilize my regression model knowing that it has strong discriminatory power?

Suppose you need to construct a procedure that makes binary decisions without human intervention. For example, the test results are reported in an automated fashion for some purpose. It is possible to find all diseased individuals (perfect TPR) by labeling everyone as diseased, but your FPR will also be 1.0. Alternatively, you could capture no false positives, but at the cost of also capturing no diseased individuals.

A ROC curve compares the tradeoffs between these two extremes, i.e. the estimated TPR and FPR for any decision value cutoff. ROC curves are commonly summarized by AUROC, but this does not imply that a model with a higher AUROC necessarily has a better TPR/FPR tradeoff at a specific decision-value.

It's common in the machine learning community to compare two or more alternative models the basis of AUROC, but this does not imply that AUROC is useful in general or even for the particular purpose of that machine learning project.

$\endgroup$
  • $\begingroup$ Despite the sites suggestion, I do feel compelled to say that this is a superb answer. The slightest quibble would be to say that, to the best of my knowledge, AUC is far more common than AUROC. Apologies if this seems overly quibblish . $\endgroup$ – aginensky Sep 19 '17 at 19:35
  • $\begingroup$ @aginensky You're correct that using AUC to mean AUROC is common practice. But users of precision-recall curves have adopted AUC as well, and the confusion between the two usages is, in my estimation, counter-productive enough to warrant the distinction. $\endgroup$ – Sycorax Sep 19 '17 at 20:39
  • 1
    $\begingroup$ I just hope that the users of precision-recall curves don't start calling it a ROC curve :) $\endgroup$ – aginensky Sep 20 '17 at 18:23
4
$\begingroup$

If the regression model gives me a subject AA with a predicted probability of 0.6 and this seems to be a high probability compared to other subjects.

Would it be correct to say that there is 85% chance that AA has the disease?

The answer is "no". The AUROC does not care about the actual value of your probability predictions, only the order of your predictions. You could divide all your prediction probabilities by 10 and still get the same AUC. In fact, you can come up with some ordering criteria that is entirely independent of what your prediction probabilities are, and still get an AUC score.

To get a good intuitive idea of how to interpret an AUROC, it helps to look at an ROC curve for a small number of samples. Here's one I whipped up: enter image description here

Note that each step to the right represents a "wrong" guess, and each step upwards represents a "right" guess. (Larger steps mean more guesses.) I filled out a grey area for a single step to the right (i.e. "wrong guess"). The AUC is simply the sum of all the dark rectangles over all wrong guesses. The height of the rectangle is the proportion of "true" samples that have been listed so far. That is, for the "false" sample individual who caused the horizontal step of the rectangle, if we stop at that individual then the true positive rate is given by the height of the rectangle. The width of the rectangle is the proportion of "false" guesses that we're running through when taking the horizontal step in the rectangle.

The area of the highlighted rectangle can be interpreted as follows. Suppose we choose a "false" sample individual at random. The probability of choosing that sample, multiplied by the true positive rate of all selections before that sample is given by the area of a dark rectangle. Thus the sum of dark rectangles is the expected true positive rate before a false sample, where the expectation is taken over all false samples. Put another way, if you pick a false sample individual at random and stop your "chosen" list at that individual, the expected value of the TPR up until that sample is the AUC.

The TPR, of course, is the probability that a positive sample, when chosen at random, will be in your "chosen" list. So another way to interpret the AUC is that, if you choose a positive sample at random, and a negative sample at random, the AUC is the expected probability that the positive sample will appear on your list before the negative sample.

As to your last question:

Based on what I said earlier, AUROC is an indicator of how well you ranked your samples, not how good your probability prediction is. Remember, any monotonic function of your probability outcomes (such as dividing by ten, or taking the sigmoid of them) will yield the exact same ROC curve. So a good AUROC shouldn't tell you how to gauge the probability of an event. (For example, if you have a very high AUROC for disease classification, and you predict that someone has a 99% chance of having disease, that person shouldn't necessarily act as if they almost definitely have a disease. It could be that they only have a 5% chance, but your model is still great at determining that their chance is much higher than that of someone else.) Because AUC is a good indicator of ranking, its main utility should be in your prioritizing candidates. For example, if you have a model with high AUC for disease classification, then your prediction results should determine who you choose first for further diagnoses or treatment.

$\endgroup$
  • $\begingroup$ I would have voted for the answer until the last sentence. If your model has poor discrimination but is better than random ($c$-index = AUROC > 0.5) and is better than other model's that other statisticians might build, then you still can use the model to help use fixed resources more wisely. And I don't find the graph as intuitive as just defining concordance based on the Wilcoxon-Mann-Whitney "all possible pairs" idea. No thresholds need be considered. $\endgroup$ – Frank Harrell Sep 19 '17 at 19:13
  • 2
    $\begingroup$ @FrankHarrell I don't understand how your comment is a refutation of my last sentence. I'm just making the point that a high AUC doesn't indicate accuracy of your prediction probabilities, just goodness of the order of probability predictions. That seems perfectly consistent with what you're saying. $\endgroup$ – Bridgeburners Sep 19 '17 at 19:20
  • $\begingroup$ I don't think so, if I understand you. The value of a probability estimate comes from having c > 0.5 and showing good calibration. But you are right that c describes the ability to correctly order probabilities. $\endgroup$ – Frank Harrell Sep 19 '17 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.