How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are there functions in R to help with this?

  • 1
    $\begingroup$ Do you mean how to construct ROC's when there are +2 models? $\endgroup$ – user30490 Aug 18 '14 at 21:23
  • $\begingroup$ Or do you mean that there are 4 outcome categories? $\endgroup$ – gung - Reinstate Monica Aug 18 '14 at 21:46
  • $\begingroup$ Categories :) I edited my post $\endgroup$ – Marcin Kosiński Aug 18 '14 at 21:51
  • $\begingroup$ I would suggest checking out this answer: stats.stackexchange.com/questions/38541/… $\endgroup$ – user30490 Aug 18 '14 at 21:58
  • 4
    $\begingroup$ What about ROC curves makes them insightful to you? Are you really interested in concordance probabilities ($c$-index; ROC area; pure discrimination measure)? I find the ROC area to be helpful even though the curves are not helpful to me. And you can generalize the idea of concordance probability to multiple categories using Somers' $D_{xy}$ rank correlation coefficient. $\endgroup$ – Frank Harrell Aug 18 '14 at 21:59

Several ideas and references are discussed in:

Other approaches include computing

  • macro-average ROC curves (average per class in a 1-vs-all fashion)
  • micro-averaged ROC curves (consider all positives and negatives together as single class)

You can see examples in some libraries like scikit-learn.

See also this other thread in CrossValidated: How to compute precision/recall for multiclass-multilabel classification?

  • $\begingroup$ FYI - the link to the multi-class ROC tutorial doesn't work $\endgroup$ – rocinante Aug 18 '14 at 22:04
  • $\begingroup$ @rocinante Fixed. $\endgroup$ – Josh Aug 18 '14 at 22:05
  • $\begingroup$ @Josh that's the vast and outstanding piece of literature :) Thank you very much. That was something I was looking for! CV is a great place. $\endgroup$ – Marcin Kosiński Aug 18 '14 at 22:28

One of the ideas is to use one-vs-all classifier. This answer gives move information about it, including some R code.

Here's a plot from that answer

enter image description here

  • $\begingroup$ Very usefull comment. Thanks for that! Nice idea by the way. $\endgroup$ – Marcin Kosiński Aug 19 '14 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.