I'm trying to do ROC analysis on my dataset.

Namely, I'm trying to find the optimal set of parameters for a predictive model. Given I just use discrete classifiers, all I have are points in the ROC space.

reading "An Introduction to ROC Analysis" by Fawcett (2006) tells that

"Informally, one point in ROC space is better than another if it is to the northwest (tp rate is higher, fp rate is lower, or both) of the first classifier."

So I guess my question is how to make the process "formal" with a sound methodology that I can back up with proper literature.

The wikipedia example makes it really obvious that given those 5 points, C' would be the best classifier. but that is done visually, and i would like automate the process.

Other questions based on my searches here on the stackexchange usually deal with the scoring-type classifiers (as pointed out again in Fawcett, 2006) and so I feel is not relevant to what I'm trying to accomplish.

Off of my pragmatic (and non-academic) thinking, I thought of:

  1. using the shortest distance to the point (0,1),

  2. using the slope of the line connecting the classifier to again the point (0,1) and using the steepest slope, supposedly to maximize TPR and minimize FPR.

but again I cannot find any literature to back these ideas.

My ROC plot by the way, looks like this:ROC plot

  • $\begingroup$ You really want a cost function... $\endgroup$ – Calimo May 16 '17 at 20:47
  • $\begingroup$ @Calimo it would seem so, stumbled upon What ROC Curves Can’t Do (and Cost Curves Can) by Drummond and Holte. are there any resources that can serve as some sort of crash-course into Cost Fxns? $\endgroup$ – carlo May 16 '17 at 21:02
  • $\begingroup$ Btw your plot doesn't look like a ROC curve, which is normally a line going from (0,0) to (1,1), increasing in a montone way. $\endgroup$ – Calimo May 17 '17 at 8:16
  • $\begingroup$ @Calimo i just plotted the results of each confusion matrix generated per simulation. isn't that to be expected from something that has a discrete classifier? from what i understand, my plot is just like the plot on wikipedia's example, only i have way more points. $\endgroup$ – carlo May 17 '17 at 8:18
  • $\begingroup$ Ok, but then that's not a ROC curve, which shows the operating performance of one classifier as the decision threshold varies (and not the model parameters). Beware of standard ROC curve analysis tools here that may just not be tailored for your problem. $\endgroup$ – Calimo May 17 '17 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.