0
$\begingroup$

I know that sensitivity and recall (and precision, and specificity, etc) can be computed for a set of data X with a binary label Y, but I'm not sure how there can be multiple values for both of those, for a single set of data.

How is an ROC curve constructed for a set of data? It must require an ordered set of data for both axes, so there must be an ordered set of pairs of recall and specificity values. How is this order determined? and how are these values determined?

$\endgroup$
  • $\begingroup$ Maybe this can be helpful? $\endgroup$ – Alexey Grigorev Dec 30 '15 at 13:43
  • $\begingroup$ That was helpful, thank you @AlexeyGrigorev, but it is still unclear to me how the "rank" or "score" is determined. That source doesn't explain that part at all. $\endgroup$ – Candic3 Dec 30 '15 at 15:31
  • 1
    $\begingroup$ A classifier is ranking if it can output some score instead of a hard classification prediction. For example, Logistic Regression typically outputs probability, which is a score you can use for ranking. You can use this information for building the ROC curve. $\endgroup$ – Alexey Grigorev Dec 30 '15 at 19:21
  • $\begingroup$ Okay, so if I have, for example, a random forest which is a binary classifier. The probability of success produced by the classifier would be the value used for ranking. Correct? $\endgroup$ – Candic3 Dec 30 '15 at 22:53
  • $\begingroup$ yes, this is correct $\endgroup$ – Alexey Grigorev Dec 31 '15 at 6:29
3
$\begingroup$

Here's an example of calculating an ROC curve. There are many things that ROC curves are used for, but this will give an overall idea of how an ROC curve is created.

Lets say that we want to use white blood cell counts to diagnose appendicitis. We'd like to collect a white blood cell count from a patient and then tell them if they have appendicitis or not. We'll call it our "Appendicitis Test". (This is just an example, no guarantees that this makes medical sense.)

We'd like to make an ROC curve for our Appendicitis Test. We have 50 patients that we can use. We know two things about these patients: 1) whether or not the patient has appendicitis and 2) what the white blood cell count of the patient is.

With these two pieces of information, we can create an ROC curve.

Here's the fake data of our patients:

#White blood cell counts for our 50 patients (WBC)
WBC = c(10:59)
#Whether they have appendicitis (Append: 1=yes, 0=no)
Append=c(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)
#Combine our data
dat = data.frame(cbind(WBC, Append))

We decide that a white blood cell count above 30 is a pretty good cutoff for deciding whether a patient has appendicitis. But we're not sure. So we try 5 possible white blood cell cutoffs that would indicate that a patient has appendicitis: 10, 20, 30, 40, and 50. Now, we need to find the sensitivity and specificity at each cutoff to see how well the test detects appendicitis at each cutoff.

#Create variables that equal 1 if the test detects 
#appendicitis (whether it's correct or not) using our five cutoffs.
dat$yes10 = ifelse(dat$WBC > 10, 1, 0)
dat$yes20 = ifelse(dat$WBC > 20, 1, 0)
dat$yes30 = ifelse(dat$WBC > 30, 1, 0)
dat$yes40 = ifelse(dat$WBC > 40, 1, 0)
dat$yes50 = ifelse(dat$WBC > 50, 1, 0)

Now calculate the sensitivity and specificity at each cutoff:

 sensitivity <-c(sum(dat$yes10 == 1 & dat$Append == 1)/sum(dat$yes10 ==1),
            sum(dat$yes20 ==1 & dat$Append == 1)/sum(dat$yes20 ==1),
            sum(dat$yes30 ==1 & dat$Append == 1)/sum(dat$yes30 ==1),
            sum(dat$yes40 ==1 & dat$Append == 1)/sum(dat$yes40 ==1),
            sum(dat$yes50 ==1 & dat$Append == 1)/sum(dat$yes50 ==1))   

 specificity <-c(sum(dat$yes10 == 0 & dat$Append == 0)/sum(dat$yes10 ==0),
                 sum(dat$yes20 ==0 & dat$Append == 0)/sum(dat$yes20 ==0),
                sum(dat$yes30 ==0 & dat$Append == 0)/sum(dat$yes30 ==0),
                sum(dat$yes40 ==0 & dat$Append == 0)/sum(dat$yes40 ==0),
                sum(dat$yes50 ==0 & dat$Append == 0)/sum(dat$yes50 ==0))

Then plot the True Positive Rate (sensitivity) by the False Positive Rate (1 - specificity) of each cutoff to get the ROC curve for the appendicitis test.

TruePositiveRate = sensitivity
FalsePositiveRate = 1 - specificity
ROCCurve = data.frame(cbind(FalsePositiveRate, TruePositiveRate))
plot(ROCCurve, main="ROC Curve for Appendicitis Test")
lines(ROCCurve, col="red")

enter image description here

Now we can see that our cutoff of 40 is our best choice, because there is a 100% true positive rate and only 20% false positive rate.

$\endgroup$
  • $\begingroup$ Good explanation. So, these "cutoff" values - it seems to me that this is just one way to vary something over multiple pairs of TPR and FPR values, as @TH4454 is suggesting. If I have multiple predictor variables, and the binary result of a machine learning algorithm, how do I incorporate all of these in devising "cutoff" values? You cited an example with just one variable as the cutoff, but it's more difficult for me to imagine if there were multiple variables.....? $\endgroup$ – Candic3 Dec 30 '15 at 5:40
  • $\begingroup$ Probabilistic/scoring classifiers, like logistic regression, trees, and neural networks, output a numeric value that needs a threshold to become a binary value. You can use an ROC plot to see which threshold is the best for your chosen probabilistic/scoring classifier. You can also look at the TPR and FPR over all of the thresholds to get an overall look at the performance of your classifier. And then you can plot an ROC curve for each type of classifier (trees, logistic, etc.) on one plot to compare all of them together and choose the best one for your data. $\endgroup$ – LindsayL Dec 31 '15 at 20:06
2
$\begingroup$

So say your model gives some % prediction between 0 and 100%. Let's call that Y. And your classifier is either A or B.

My understanding is that an ROC curve is built by ranging a variable, let's call it "K", for values between 0 and 100%. For every value K and every estimate Y, you say something like ... if K is greater than Y, then it's an A, otherwise it's a B.

Then you build a confusion matrix for every K. From each of those confusion matrices, you can determine the true positive (the Y value in the ROC curve) and false positive (X value in the ROC curve).

Hence, if you have perfect separation between A and B, your ROC curve will go straight up and to the left.

$\endgroup$
1
$\begingroup$

A ROC Curve is not constructed for a set of data, it is constructed for the results of a classification performed on a set of data.

There are models (or methods of implementing them) that produce multiple ROC curves for a single model and set- say, one for the results of the model applied to the training set itself and one for the results of the model applied to validation set. Multiple ROC curves for a single model and data set would most likely be of this variety.

$\endgroup$
1
$\begingroup$

A ROC curve is calculated from an independent risk prediction or risk score that has been merged to validation data containing observed binary outcome variables, e.g. life or death, recurrence or remission, guilty or innocent, etc..

The ranges of possible values for that risk prediction/score are sorted and enumerated from least to greatest as a range of possible cutpoints. For each cutpoint, the continuous risk variable is dichotomized producing a classification of the observed event. This binary value is compared to the observed outcomes and the sensitivity and specificity is calculated each time. Plotting sensitivity and specificity over the range of possible cutpoints produces a ROC.

Note that the linepath produced by these cutpoints does not convey what the actual cutpoint is. For that reason, it is often nice to annotate a ROC curve with example sens/spec combinations for a couple cutoff points.

Example R code here:

ROC<-function(T,D){
  DD <- table(-T,D)
  tpr <- cumsum(DD[,2])/sum(DD[,2])
  fpr <- cumsum(DD[,1])/sum(DD[,1])
  rval <- list(tpr=tpr, fpr=fpr,
  cutpoints=rev(sort(unique(T))),
  call=sys.call())
  class(rval)<-"ROC"
  rval
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.