Here's an example of calculating an ROC curve. There are many things that ROC curves are used for, but this will give an overall idea of how an ROC curve is created.
Lets say that we want to use white blood cell counts to diagnose appendicitis. We'd like to collect a white blood cell count from a patient and then tell them if they have appendicitis or not. We'll call it our "Appendicitis Test". (This is just an example, no guarantees that this makes medical sense.)
We'd like to make an ROC curve for our Appendicitis Test. We have 50 patients that we can use. We know two things about these patients: 1) whether or not the patient has appendicitis and 2) what the white blood cell count of the patient is.
With these two pieces of information, we can create an ROC curve.
Here's the fake data of our patients:
#White blood cell counts for our 50 patients (WBC)
WBC = c(10:59)
#Whether they have appendicitis (Append: 1=yes, 0=no)
Append=c(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)
#Combine our data
dat = data.frame(cbind(WBC, Append))
We decide that a white blood cell count above 30 is a pretty good cutoff for deciding whether a patient has appendicitis. But we're not sure. So we try 5 possible white blood cell cutoffs that would indicate that a patient has appendicitis: 10, 20, 30, 40, and 50. Now, we need to find the sensitivity and specificity at each cutoff to see how well the test detects appendicitis at each cutoff.
#Create variables that equal 1 if the test detects
#appendicitis (whether it's correct or not) using our five cutoffs.
dat$yes10 = ifelse(dat$WBC > 10, 1, 0)
dat$yes20 = ifelse(dat$WBC > 20, 1, 0)
dat$yes30 = ifelse(dat$WBC > 30, 1, 0)
dat$yes40 = ifelse(dat$WBC > 40, 1, 0)
dat$yes50 = ifelse(dat$WBC > 50, 1, 0)
Now calculate the sensitivity and specificity at each cutoff:
sensitivity <-c(sum(dat$yes10 == 1 & dat$Append == 1)/sum(dat$yes10 ==1),
sum(dat$yes20 ==1 & dat$Append == 1)/sum(dat$yes20 ==1),
sum(dat$yes30 ==1 & dat$Append == 1)/sum(dat$yes30 ==1),
sum(dat$yes40 ==1 & dat$Append == 1)/sum(dat$yes40 ==1),
sum(dat$yes50 ==1 & dat$Append == 1)/sum(dat$yes50 ==1))
specificity <-c(sum(dat$yes10 == 0 & dat$Append == 0)/sum(dat$yes10 ==0),
sum(dat$yes20 ==0 & dat$Append == 0)/sum(dat$yes20 ==0),
sum(dat$yes30 ==0 & dat$Append == 0)/sum(dat$yes30 ==0),
sum(dat$yes40 ==0 & dat$Append == 0)/sum(dat$yes40 ==0),
sum(dat$yes50 ==0 & dat$Append == 0)/sum(dat$yes50 ==0))
Then plot the True Positive Rate (sensitivity) by the False Positive Rate (1 - specificity) of each cutoff to get the ROC curve for the appendicitis test.
TruePositiveRate = sensitivity
FalsePositiveRate = 1 - specificity
ROCCurve = data.frame(cbind(FalsePositiveRate, TruePositiveRate))
plot(ROCCurve, main="ROC Curve for Appendicitis Test")
lines(ROCCurve, col="red")

Now we can see that our cutoff of 40 is our best choice, because there is a 100% true positive rate and only 20% false positive rate.
random forest
which is a binary classifier. The probability of success produced by the classifier would be the value used for ranking. Correct? $\endgroup$