How is an ROC curve constructed for a set of data?

Question

I know that sensitivity and recall (and precision, and specificity, etc) can be computed for a set of data X with a binary label Y, but I'm not sure how there can be multiple values for both of those, for a single set of data.

How is an ROC curve constructed for a set of data? It must require an ordered set of data for both axes, so there must be an ordered set of pairs of recall and specificity values. How is this order determined? and how are these values determined?

Answer 1

Here's an example of calculating an ROC curve. There are many things that ROC curves are used for, but this will give an overall idea of how an ROC curve is created.

Lets say that we want to use white blood cell counts to diagnose appendicitis. We'd like to collect a white blood cell count from a patient and then tell them if they have appendicitis or not. We'll call it our "Appendicitis Test". (This is just an example, no guarantees that this makes medical sense.)

We'd like to make an ROC curve for our Appendicitis Test. We have 50 patients that we can use. We know two things about these patients: 1) whether or not the patient has appendicitis and 2) what the white blood cell count of the patient is.

With these two pieces of information, we can create an ROC curve.

Here's the fake data of our patients:

#White blood cell counts for our 50 patients (WBC)
WBC = c(10:59)
#Whether they have appendicitis (Append: 1=yes, 0=no)
Append=c(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)
#Combine our data
dat = data.frame(cbind(WBC, Append))

We decide that a white blood cell count above 30 is a pretty good cutoff for deciding whether a patient has appendicitis. But we're not sure. So we try 5 possible white blood cell cutoffs that would indicate that a patient has appendicitis: 10, 20, 30, 40, and 50. Now, we need to find the sensitivity and specificity at each cutoff to see how well the test detects appendicitis at each cutoff.

#Create variables that equal 1 if the test detects 
#appendicitis (whether it's correct or not) using our five cutoffs.
dat$yes10 = ifelse(dat$WBC > 10, 1, 0)
dat$yes20 = ifelse(dat$WBC > 20, 1, 0)
dat$yes30 = ifelse(dat$WBC > 30, 1, 0)
dat$yes40 = ifelse(dat$WBC > 40, 1, 0)
dat$yes50 = ifelse(dat$WBC > 50, 1, 0)

Now calculate the sensitivity and specificity at each cutoff:

 sensitivity <-c(sum(dat$yes10 == 1 & dat$Append == 1)/sum(dat$yes10 ==1),
            sum(dat$yes20 ==1 & dat$Append == 1)/sum(dat$yes20 ==1),
            sum(dat$yes30 ==1 & dat$Append == 1)/sum(dat$yes30 ==1),
            sum(dat$yes40 ==1 & dat$Append == 1)/sum(dat$yes40 ==1),
            sum(dat$yes50 ==1 & dat$Append == 1)/sum(dat$yes50 ==1))   

 specificity <-c(sum(dat$yes10 == 0 & dat$Append == 0)/sum(dat$yes10 ==0),
                 sum(dat$yes20 ==0 & dat$Append == 0)/sum(dat$yes20 ==0),
                sum(dat$yes30 ==0 & dat$Append == 0)/sum(dat$yes30 ==0),
                sum(dat$yes40 ==0 & dat$Append == 0)/sum(dat$yes40 ==0),
                sum(dat$yes50 ==0 & dat$Append == 0)/sum(dat$yes50 ==0))

Then plot the True Positive Rate (sensitivity) by the False Positive Rate (1 - specificity) of each cutoff to get the ROC curve for the appendicitis test.

TruePositiveRate = sensitivity
FalsePositiveRate = 1 - specificity
ROCCurve = data.frame(cbind(FalsePositiveRate, TruePositiveRate))
plot(ROCCurve, main="ROC Curve for Appendicitis Test")
lines(ROCCurve, col="red")

Now we can see that our cutoff of 40 is our best choice, because there is a 100% true positive rate and only 20% false positive rate.

Answer 2

A ROC curve is calculated from an independent risk prediction or risk score that has been merged to validation data containing observed binary outcome variables, e.g. life or death, recurrence or remission, guilty or innocent, etc..

The ranges of possible values for that risk prediction/score are sorted and enumerated from least to greatest as a range of possible cutpoints. For each cutpoint, the continuous risk variable is dichotomized producing a classification of the observed event. This binary value is compared to the observed outcomes and the sensitivity and specificity is calculated each time. Plotting sensitivity and specificity over the range of possible cutpoints produces a ROC.

Note that the linepath produced by these cutpoints does not convey what the actual cutpoint is. For that reason, it is often nice to annotate a ROC curve with example sens/spec combinations for a couple cutoff points.

Example R code here:

ROC<-function(T,D){
  DD <- table(-T,D)
  tpr <- cumsum(DD[,2])/sum(DD[,2])
  fpr <- cumsum(DD[,1])/sum(DD[,1])
  rval <- list(tpr=tpr, fpr=fpr,
  cutpoints=rev(sort(unique(T))),
  call=sys.call())
  class(rval)<-"ROC"
  rval
}

Answer 3

So say your model gives some % prediction between 0 and 100%. Let's call that Y. And your classifier is either A or B.

My understanding is that an ROC curve is built by ranging a variable, let's call it "K", for values between 0 and 100%. For every value K and every estimate Y, you say something like ... if K is greater than Y, then it's an A, otherwise it's a B.

Then you build a confusion matrix for every K. From each of those confusion matrices, you can determine the true positive (the Y value in the ROC curve) and false positive (X value in the ROC curve).

Hence, if you have perfect separation between A and B, your ROC curve will go straight up and to the left.

Answer 4

A ROC Curve is not constructed for a set of data, it is constructed for the results of a classification performed on a set of data.

There are models (or methods of implementing them) that produce multiple ROC curves for a single model and set- say, one for the results of the model applied to the training set itself and one for the results of the model applied to validation set. Multiple ROC curves for a single model and data set would most likely be of this variety.