How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are there functions in R to help with this?
$\begingroup$
$\endgroup$
11
-
1$\begingroup$ Do you mean how to construct ROC's when there are +2 models? $\endgroup$– user30490Aug 18, 2014 at 21:23
-
$\begingroup$ Or do you mean that there are 4 outcome categories? $\endgroup$– gung - Reinstate MonicaAug 18, 2014 at 21:46
-
$\begingroup$ Categories :) I edited my post $\endgroup$– MarcinAug 18, 2014 at 21:51
-
$\begingroup$ I would suggest checking out this answer: stats.stackexchange.com/questions/38541/… $\endgroup$– user30490Aug 18, 2014 at 21:58
-
4$\begingroup$ What about ROC curves makes them insightful to you? Are you really interested in concordance probabilities ($c$-index; ROC area; pure discrimination measure)? I find the ROC area to be helpful even though the curves are not helpful to me. And you can generalize the idea of concordance probability to multiple categories using Somers' $D_{xy}$ rank correlation coefficient. $\endgroup$– Frank HarrellAug 18, 2014 at 21:59
|
Show 6 more comments
2 Answers
$\begingroup$
$\endgroup$
3
Several ideas and references are discussed in:
- A simple generalization of the area under the ROC curve to multiple class classification problems.
- Multi-class ROC (a tutorial) (using "volumes" under ROC)
Other approaches include computing
- macro-average ROC curves (average per class in a 1-vs-all fashion)
- micro-averaged ROC curves (consider all positives and negatives together as single class)
You can see examples in some libraries like scikit-learn.
See also this other thread in CrossValidated: How to compute precision/recall for multiclass-multilabel classification?
-
$\begingroup$ FYI - the link to the multi-class ROC tutorial doesn't work $\endgroup$ Aug 18, 2014 at 22:04
-
-
$\begingroup$ @Josh that's the vast and outstanding piece of literature :) Thank you very much. That was something I was looking for! CV is a great place. $\endgroup$– MarcinAug 18, 2014 at 22:28
$\begingroup$
$\endgroup$
1
One of the ideas is to use one-vs-all classifier. This answer gives move information about it, including some R code.
Here's a plot from that answer
answered Aug 19, 2014 at 13:14
-
$\begingroup$ Very usefull comment. Thanks for that! Nice idea by the way. $\endgroup$– MarcinAug 19, 2014 at 13:50