Manual construction of ROC rurve

Question

I have read several articles about how to build a ROC curve but there is something I do not understand. Assuming I have a binary classifier, to draw a ROC curve I have to retrieve a list of scores (let's say long N) and for every entry of such a list the quantities trp and fpr. My question is: how do I get this list of scores? Do I train my model N different times using N different training sets and test it against the same testing set? Do I train the model once and I test it N times using N different testing sets?

Also, once I got the three vectors (scores, tprs and fprs), should I sort them such that the fpr vector is in ascending order?

Answer 1

  # in R software  
library(ROCR)
data(ROCR.simple) # your model (result of glm for example)
pred <- prediction( ROCR.simple$predictions, ROCR.simple$labels)
perf <- performance(pred,"sens","spec")
plot(perf) # Plot ROC Curve

dcutoff=data.frame() 

for (cutoff in seq(0, 1, 0.01)) {
indice = which.min(abs([email protected][[1]] - cutoff)) 
sensitivity = [email protected][[1]][indice] 
specificity = [email protected][[1]][indice]
total=sensitivity+specificity

dcutoff=rbind(dcutoff, data.frame(cutoff, sensitivity, specificity, total))
}

# return : 1 - dcutoff, a data frame with sens and spec by cutoffs
#          2- best_cutoff, the best cut off that maximizes sens+spec
best_cutoff = dcutoff[ which.max(dcutoff$total) , c("cutoff")]