How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are there functions in R to help with this?
$\begingroup$
$\endgroup$
gung - Reinstate Monica
125k42 gold badges331 silver badges618 bronze badges
asked Aug 18 '14 at 21:17
-
1$\begingroup$ Do you mean how to construct ROC's when there are +2 models? $\endgroup$ – user30490 Aug 18 '14 at 21:23
-
$\begingroup$ Or do you mean that there are 4 outcome categories? $\endgroup$ – gung - Reinstate Monica Aug 18 '14 at 21:46
-
$\begingroup$ Categories :) I edited my post $\endgroup$ – Marcin Kosiński Aug 18 '14 at 21:51
-
$\begingroup$ I would suggest checking out this answer: stats.stackexchange.com/questions/38541/… $\endgroup$ – user30490 Aug 18 '14 at 21:58
-
4$\begingroup$ What about ROC curves makes them insightful to you? Are you really interested in concordance probabilities ($c$-index; ROC area; pure discrimination measure)? I find the ROC area to be helpful even though the curves are not helpful to me. And you can generalize the idea of concordance probability to multiple categories using Somers' $D_{xy}$ rank correlation coefficient. $\endgroup$ – Frank Harrell Aug 18 '14 at 21:59
$\begingroup$
$\endgroup$
Several ideas and references are discussed in:
- A simple generalization of the area under the ROC curve to multiple class classification problems.
- Multi-class ROC (a tutorial) (using "volumes" under ROC)
Other approaches include computing
- macro-average ROC curves (average per class in a 1-vs-all fashion)
- micro-averaged ROC curves (consider all positives and negatives together as single class)
You can see examples in some libraries like scikit-learn.
See also this other thread in CrossValidated: How to compute precision/recall for multiclass-multilabel classification?
-
$\begingroup$ FYI - the link to the multi-class ROC tutorial doesn't work $\endgroup$ – rocinante Aug 18 '14 at 22:04
-
-
$\begingroup$ @Josh that's the vast and outstanding piece of literature :) Thank you very much. That was something I was looking for! CV is a great place. $\endgroup$ – Marcin Kosiński Aug 18 '14 at 22:28
$\begingroup$
$\endgroup$
One of the ideas is to use one-vs-all classifier. This answer gives move information about it, including some R code.
Here's a plot from that answer
answered Aug 19 '14 at 13:14
-
$\begingroup$ Very usefull comment. Thanks for that! Nice idea by the way. $\endgroup$ – Marcin Kosiński Aug 19 '14 at 13:50
