4
$\begingroup$

Currently I'm asking me how to draw the ROC curve (Receiver Operating Characteristic curve). On the x-axis there is the false positive rate (1-specificity) and on the y-axis there is the true positive rate (sensitivity). I'm using nested cross-validation. If I evaluate the model I'm getting one point of the ROC curve.

I have looked it up on Google and I found two ways:

  1. Evaluating the model for many paramter combinations (model tuning). For each parameter combination there is one point on the ROC curve.

  2. For each probability cutoff point there is one point on the ROC curve.

Which one is the right one or is there another way to do it?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
5
$\begingroup$

An ROC curve visualizes the performance of a model in different configurations (= cutoffs), and hence the second option is the right way.

In the first option you are somehow plotting points of different models (same learning approach with different hyperparameters), which is not related to ROC curves. In fact, which points would you even plot of these different models to make them somewhat calibrated and comparable? All points with $P(\hat{Y}=1) = 0.5$?

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for the answer. Yes I thought to plot all points which P(Y=1) = 0.5 and doing this for all possible parameters combinations. However, could you perhaps provide an explanation how to derive these different configurations, that mean the different probability cutoffs? A lot of techniques do not produce probability output. $\endgroup$
    – machinery
    Jan 1, 2016 at 15:43
  • $\begingroup$ By the way, I want to derive the ROC curve in Matlab. $\endgroup$
    – machinery
    Jan 1, 2016 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.