Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 14 down vote favorite
4

In other words, instead of having a two class problem I am dealing with 4 classes and still would like to assess performance using AUC.

share|improve this question
Is dimension reduction an option? – Shane Aug 27 '10 at 1:59
we have 4 classes, not really – CLOCK Aug 27 '10 at 3:03

4 Answers

up vote 16 down vote

It seems you are looking for multi-class ROC analysis, which is a kind of multi-objective optimization covered in a tutorial at ICML'04. As in several multi-class problem, the idea is generally to carry out pairwise comparison (one class vs. all other classes, one class vs. another class, see (1) or the Elements of Statistical Learning), and there is a recent paper by Landgrebe and Duin on that topic, Approximating the multiclass ROC by pairwise analysis, Pattern Recognition Letters 2007 28: 1747-1758. Now, for visualization purpose, I've seen some papers some time ago, most of them turning around volume under the ROC surface (VUS) or Cobweb diagram.

I don't know, however, if there exists an R implementation of these methods, although I think the stars() function might be used for cobweb plot. I just ran across a Matlab toolbox that seems to offer multi-class ROC analysis, PRSD Studio.

Other papers that may also be useful as a first start for visualization/computation:

References:
1. Allwein, E.L., Schapire, R.E. and Singer, Y. (2000). Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of Machine Learning Research, 1:113–141.

share|improve this answer
1  
THIS IS FANTASTIC THANK YOU – CLOCK Aug 27 '10 at 15:03
up vote 4 down vote

While the math is beyond me this general review article has some references you will likely be interested in, and has a brief description of multi-class ROC graphs.

An introduction to ROC analysis by Tom Fawcett Pattern Recognition Letters Volume 27, Issue 8, June 2006, Pages 861-874

Link to pdf as provided by gd047- thanks

share|improve this answer
2  
I would say the same providing another link google.gr/… – gd047 Aug 27 '10 at 6:40
1  
And here is another one, directly related to multi-class problem: Multi-class ROC analysis from a multi-objective optimisation perspective, Pattern Recognition Letters 2006 27(8): 918-927 (j.mp/9AMgzq). – chl♦ Aug 27 '10 at 7:38
1  
Thanks for the other links, apparently that whole issue is popular, and its archive on science direct can be found here sciencedirect.com/… – Andy W Aug 27 '10 at 13:17
up vote 0 down vote

I recently found this pROC package in R which plots a multiclass ROC using the technique specified by Hand and Till (2001). You can use the multiclass.roc function.

share|improve this answer
up vote 0 down vote

Weka does this perfectly, using the one against all class strategy.

share|improve this answer
1  
Do you perhaps have a specific pointer on how to perform multi-class ROC with Weka? – chl♦ Sep 29 '12 at 8:22
This is more of a comment than an answer, as it stands. Usually answers offer more detail. Welcome to the site! – Peter Flom Sep 30 '12 at 11:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.