1
$\begingroup$

I am doing an experiments where changing a parameter I am obtaining different number of FalsePositive, FalseNegative... and so on. I am using this parameter tuning as threshold tuning to obtain FPR and TPR to build a ROC curve. However, the problem is so easy tha it gets most of the time the point near [1,0]. I never have any point near [1,1]. If I want to compute the AUC, what am I supposed to do? To add artificially a point [1,1]?

Moreover, for another experiment I have some values of TPR and FPR which are disproportionated to other cases, and they look weird.

Thank you. Best,enter image description here


Thank you. Now I am using directly probabilities from the classifier. However, I am still not getting to the point [1,1].

$\endgroup$
  • $\begingroup$ It's fine to add points at (0,0) and (1,1) to a ROC curve, but what you have here cannot be a ROC curve because ROC curves are defined to be non-decreasing! $\endgroup$ – Sycorax Nov 28 '17 at 21:36
  • $\begingroup$ @sycorax missing those points means that you are missing thresholds. In my experience it is usually indicative of a problem deeper than those two points. $\endgroup$ – Calimo Nov 29 '17 at 7:12
  • $\begingroup$ @Alex how are you getting your thresholds? Are you for instance taking random points in the interval (0,1)? $\endgroup$ – Calimo Nov 29 '17 at 7:21
  • 1
    $\begingroup$ @Calimo We're saying the same thing. If the max and min prediction values on some data set are 0.9 and 0.1, then the ROC points (0,0) and (1,1) are generated by appending predictions smaller and larger than 0.1 and 0.9, resp. and then proceeding as usual with cumulative sums or whatnot. Another way to look at this is that a ROC curve is characterized as a nondecreasing curve from (0,0) to (1,1), so appending (0,0) and (1,1) has no significance beyond satisfying this quality. $\endgroup$ – Sycorax Nov 29 '17 at 14:41
3
$\begingroup$

To create a ROC curve, you need to vary the decision threshold of a single classifier, not the parameters of the model itself.

I recommend Tom Fawcett's "An Introduction to ROC Analysis" to get started with ROC analysis. He gives an algorithm to correctly build the full ROC curve that always contains all thresholds, including (0,0) and (1,1) (see Algorithm 1, p. 866).

$\endgroup$
  • $\begingroup$ Thank you. Now I am using directly probabilities from the classifier. However, I am still not getting to the point [1,1]. $\endgroup$ – Alex Nov 29 '17 at 0:40
  • $\begingroup$ @alex you should have posted a new question, this one is completely different. But really you should read Fawcett's paper. Algorithm 1 gives you a curve that always contains (1,1). I edited my answer to reflect this. $\endgroup$ – Calimo Nov 29 '17 at 7:00
1
$\begingroup$

If Calimo's answer has helped you generate a legitimate ROC curve, it's OK to manually add a point at (0, 0) or (1, 1) if you're trying to calculate the AUC by summing up the areas of rectangles.

Note that there are other interpretations of AUC that don't invoke the ROC curve at all, e.g. a concordance measure. See the accepted answere here for more detail.

Finally, I think your original question might be reflecting a misunderstanding of how the ROC curve is constructed; in addition to Tom Fawcett's paper that Calimo linked to, a while ago I put together a visual introduction that you or other visitors might find helpful:

https://callumwebby.com/portfolio/ROC/

I haven't got around to discussing AUC, but it includes some example code that might help understanding. You'll see that I manually add a point at (0, 0) just so the endpoints span the full space.

$\endgroup$
  • $\begingroup$ The absence of (1,1) indicates that he is still not doing a ROC curve. A good algorithm such as the one described by Fawcett gives you that point consistently 100% of the time. $\endgroup$ – Calimo Nov 29 '17 at 7:04
  • 1
    $\begingroup$ Having one of the endpoints missing can easily just be a result of only plotting points in the ROC space for the n predictions when you really need n + 1. It doesn't invalidate the rest of the curve. The algorithm described by Fawcett avoids this by creating a point at (0, 0) before considering the predictions, similar to my cumulative sum approach which appends a pair of zeros to the result. $\endgroup$ – Callum Webb Nov 29 '17 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.