# receiver operating characteristic (ROC) on a test set

The following image definitely makes sense to me.

Say you have a few trained binary classifiers A, B (B not much better than random guessing etc. ...) and a test set composed of n test samples to go with all those classifiers. Since Precision and Recall are computed for all n samples, those dots corresponding to classifiers make sense.

Now sometimes people talk about ROC curves and I understand that precision is expressed as a function of recall or simply plotted Precision(Recall).

I don't understand where does this variability come from, since you have a fixed number of test samples. Do you just pick some subsets of the test set and find precision and recall in order to plot them and hence many discrete values (or an interpolated line) ?

• Are you asking why it's a curve instead of a single point? – Tchotchke Aug 13 '15 at 18:25
• Yes. precisely, I mean for the whole dataset, say an SVM classifier gives one recall value and one precision. – valentin Aug 13 '15 at 18:31
• @valentin: maybe this can help ? stats.stackexchange.com/questions/166987/… – user83346 Aug 14 '15 at 8:51

You get different sensitivity/ specificity pairs for every treshhold of your marker variable. If you plot this pairs as points in the ROC plot and connect the points with lines you get the ROC curve.

Explanation with an example

Let's assume you have some binary outcome variable $$y$$ and you want to classify your cases to $$y$$ using some marker variable $$x$$. Here is an other vizualitation than the usual one:

What happens if you set the treshhold at -1? See the following plot:

Obviously, this will classify all cases with $$y= 1$$ correctly, while many cases with $$y= 0$$ are misclassified. What if you choose another treshhold, say -.5?

Here, the red line at -.5 shows that a few cases with $$y=1$$ and quite some cases with $$y= 0$$ are misclassified. This means, that for every treshhold that you choose you will get different fractions of (in)correctly classified cases of $$y= 1$$ and $$y= 0$$. Since the fraction of correctly classified positive cases ($$y= 1$$) is called sensitivity and the fraction of correctly classified false cases ($$y= 0$$) is called specificity, it is obvious that you get different sensitivity/ specificity pairs for every treshhold of your marker variable. If you plot this pairs as points in the ROC plot and connect the points with lines you get the ROC curve.

R code

# generate data like here https://stats.stackexchange.com/questions/46523/how-to-simulate-artificial-data-for-logistic-regression/46525
set.seed(666)
x = rnorm(1000)           # some continuous variables
z = 1 + 7*x        # linear combination with a bias
pr = 1/(1+exp(-z))         # pass through an inv-logit function
y = rbinom(1000,1,pr)

# make a plot
plot(x, y)
abline(v= -1)
abline(v= -.5, col= "red")


If you're looking for an explanation of ROC, An Introduction to Statistical Learning, can provide a good, brief overview (pages 147-148). Basically, you'll get different True Positive and False Positive rates as you value the threshold value for what you determines a positive.

For a more detailed discussion of model evaluation, I really like the presentation in Applied Predictive Modeling.

Often procedures will have some kind of tuning parameter or cutoff value that you can vary. At different values of this parameter the procedure will produce different sensitivity/specificity. It is this that is plotted by ROC curve

For example in logistic regression where you are trying to do binary classification, the final outcome is a value between 0 and 1. What cutoff should you use to put a value into one category or another? 0.5 seems like a natural choice, but is this the best?

• I don't think that's quite right - the ROC curve is only for one set of values for a parameter (e.g., cost = 10 in a Linear SVM) - the curve comes solely from the different cutoffs that you set. The ROC curve can help you determine which set of parameters to use or which cutoff is best for your particular situation. – Tchotchke Aug 13 '15 at 19:33
• Just as a mental exercise, say the classifier is fixed and the threshold is in place (e.g. svm), if you start exploring your test set in small distinct chunks with such a hypothesis, and plot (precision, 1-recall) points each time... I wonder ... should it resemble a discrete ROC curve, after all the relationship between precision and recall is intrinsic to the classifier – valentin Aug 13 '15 at 20:01
• @Tchotchke not exactly sure to which part you are saying "I don't think that's quite right". Maybe to my use of the word "tuning" parameter? I I had the cutoff value in mind when I was thinking of that. – bdeonovic Aug 13 '15 at 20:04
• Right - "tuning parameter" refers to a characteristic of the algorithm and is distinct from the cutoff value, so I think it's confusing (and wrong, in this context) to conflate the two. – Tchotchke Aug 13 '15 at 20:07
• With a fully parametrised SVM there is no direct scoring or probability output, just the class label, does that mean there's no ROC curve for it ? Or you can explore various parts of the test set and plot the relationship (precision, 1-recall). What confused me in the first place was the fact that you could compute the average interpolated precision using 11 levels of recall as though you could somehow obtain them from a fully specified SVM – valentin Aug 13 '15 at 20:15