2
$\begingroup$

I am doing an experiments where changing a parameter I am obtaining different number of FalsePositive, FalseNegative... and so on. I am using this parameter tuning as threshold tuning to obtain FPR and TPR to build a ROC curve. However, the problem is so easy tha it gets most of the time the point near [1,0]. I never have any point near [1,1]. If I want to compute the AUC, what am I supposed to do? To add artificially a point [1,1]?

Moreover, for another experiment I have some values of TPR and FPR which are disproportionated to other cases, and they look weird.

Thank you. Best,enter image description here

Thank you. Now I am using directly probabilities from the classifier. However, I am still not getting to the point [1,1].

Improve this question
$\endgroup$
4
  • $\begingroup$ It's fine to add points at (0,0) and (1,1) to a ROC curve, but what you have here cannot be a ROC curve because ROC curves are defined to be non-decreasing! $\endgroup$
    – Sycorax ♦
    Commented Nov 28, 2017 at 21:36
  • $\begingroup$ @sycorax missing those points means that you are missing thresholds. In my experience it is usually indicative of a problem deeper than those two points. $\endgroup$
    – Calimo
    Commented Nov 29, 2017 at 7:12
  • $\begingroup$ @Alex how are you getting your thresholds? Are you for instance taking random points in the interval (0,1)? $\endgroup$
    – Calimo
    Commented Nov 29, 2017 at 7:21
  • 1
    $\begingroup$ @Calimo We're saying the same thing. If the max and min prediction values on some data set are 0.9 and 0.1, then the ROC points (0,0) and (1,1) are generated by appending predictions smaller and larger than 0.1 and 0.9, resp. and then proceeding as usual with cumulative sums or whatnot. Another way to look at this is that a ROC curve is characterized as a nondecreasing curve from (0,0) to (1,1), so appending (0,0) and (1,1) has no significance beyond satisfying this quality. $\endgroup$
    – Sycorax ♦
    Commented Nov 29, 2017 at 14:41

3 Answers 3

Reset to default
4
$\begingroup$

To create a ROC curve, you need to vary the decision threshold of a single classifier, not the parameters of the model itself.

I recommend Tom Fawcett's "An Introduction to ROC Analysis" to get started with ROC analysis. He gives an algorithm to correctly build the full ROC curve that always contains all thresholds, including (0,0) and (1,1) (see Algorithm 1, p. 866).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. Now I am using directly probabilities from the classifier. However, I am still not getting to the point [1,1]. $\endgroup$
    – Alex
    Commented Nov 29, 2017 at 0:40
  • $\begingroup$ @alex you should have posted a new question, this one is completely different. But really you should read Fawcett's paper. Algorithm 1 gives you a curve that always contains (1,1). I edited my answer to reflect this. $\endgroup$
    – Calimo
    Commented Nov 29, 2017 at 7:00
1
$\begingroup$

If Calimo's answer has helped you generate a legitimate ROC curve, it's OK to manually add a point at (0, 0) or (1, 1) if you're trying to calculate the AUC by summing up the areas of rectangles.

Note that there are other interpretations of AUC that don't invoke the ROC curve at all, e.g. a concordance measure. See the accepted answere here for more detail.

Finally, I think your original question might be reflecting a misunderstanding of how the ROC curve is constructed; in addition to Tom Fawcett's paper that Calimo linked to, a while ago I put together a visual introduction that you or other visitors might find helpful:

https://callumwebby.com/portfolio/ROC/

I haven't got around to discussing AUC, but it includes some example code that might help understanding. You'll see that I manually add a point at (0, 0) just so the endpoints span the full space.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The absence of (1,1) indicates that he is still not doing a ROC curve. A good algorithm such as the one described by Fawcett gives you that point consistently 100% of the time. $\endgroup$
    – Calimo
    Commented Nov 29, 2017 at 7:04
  • 1
    $\begingroup$ Having one of the endpoints missing can easily just be a result of only plotting points in the ROC space for the n predictions when you really need n + 1. It doesn't invalidate the rest of the curve. The algorithm described by Fawcett avoids this by creating a point at (0, 0) before considering the predictions, similar to my cumulative sum approach which appends a pair of zeros to the result. $\endgroup$
    – Callum Webb
    Commented Nov 29, 2017 at 9:34
0
$\begingroup$

Ok, it looks to me like, as you say, your task is far to easy and you are achieving a really high d' with maybe a change in variance. However, it is hard to say unless you actually fit an unequal-variant SDT model to your empirical data. Honestly, you are probably never going to get enough power in your experiment to go to [1,1]. You really should reduce the level of discriminability between your target and noise stimuli to remove the ceiling effect. I really can't see anything wrong with calculating AUC from what you have already. But you should really adjust your stimuli (e.g. make your noise stimuli more similar to the target stimuli - or degrade the target stimuli) and redo the experiment. You should notice a reduction in AUC and the data reaching a perfect detection further towards [1,1].

Improve this answer
$\endgroup$
1
  • $\begingroup$ I don't understand how this answers the question. $\endgroup$
    – Michael R. Chernick
    Commented Sep 9, 2019 at 3:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.