How to draw ROC curve with three response variable? [duplicate]

Question

I have a toy example reproduced below in which the response variable has three possible classes. I am trying to create an ROC but not sure how to deal with it when there are three classes. Any help will be appreciated. Thanks

library(ipred)
control = rpart.control(maxdepth = 20, minsplit = 20, cp = 0.01, maxsurrogate=2, surrogatestyle = 0, xval=25)
n <- 500; p <- 10
f <- function(x,a,b,d) return( a*(x-b)^2+d )
x1 <- runif(n/2,0,4)
y1 <- f(x1,-1,2,1.7)+runif(n/2,-1,1)
x2 <- runif(n/2,2,6)
y2 <- f(x2,1,4,-1.7)+runif(n/2,-1,1)
y <- c(rep(-1,floor(n/3)),rep(0,ceiling(n/3)), rep(1,ceiling(n/3)))
dat <- data.frame(y=factor(y),x1=c(x1,x2),x2=c(y1,y2), matrix(rnorm(n*(p-2)),ncol=(p-2)))
names(dat)<-c("y",paste("x",1:p,sep=""))
dat

plot(dat$x1,dat$x2,pch=c(1:2)[y], col=c(1,8)[y], 
     xlab=names(dat)[2],ylab=names(dat)[3])
indtrain<-sample(1:n,300,replace=FALSE)
train<-dat[indtrain,]; dim(train) 
test<-dat[setdiff(1:n,indtrain),]; dim(test) 
test

mod <- bagging(y~.,  data=train, control=control, coob=TRUE, nbagg=25, keepX = TRUE)
mod
pred<-predict(mod, newdata=test[,-1],type="prob", aggregation= "average"); pred

For two class case, I use to do the following but it is no longer valid for three classes.

yhat <- pred[,2]
y = test[, -1]
plot.roc(y, yhat)

Answer 1

You might want to have a look at the Volume Under the ROC Surface as defined in the following articles:

• Ferri C, Hernández-orallo J, Salido MA. Volume Under the ROC Surface for Multi-class Problems. Exact Computation and Evaluation of Approximations. PROC OF 14TH EUROPEAN CONFERENCE ON MACHINE LEARNING. 2003;108‑120.
• He X, Frey EC. The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS). IEEE Transactions on Medical Imaging. 2008;27(5):577‑588.

Answer 2

ROC Analysis was designed for dealing with only two variables: noise and no noise, so using it for 3 or more variables makes little sense.

However, you for any multi-classification problem it's possible to use a bunch of binary classifiers and do so-called One-Vs-All Classification

E.g. consider the IRIS data set: there are 3 classes: setosa, versicolor, and virginica. So we can build 3 classifiers (e.g. Naive Bayes): for setosa, for vesicolor and for virginica. And then draw a ROC curve for each and tune the threshold for each model separately. AUC in such a case could be just the average across AUCs for individual models.

Here's a ROC curve for the IRIS data set:

AUC in this case is $\approx 0.98 = \frac{1 + 0.98 + 0.97}{3}$

R Code:

library(ROCR)
library(klaR)

data(iris)

lvls = levels(iris$Species)
testidx = which(1:length(iris[, 1]) %% 5 == 0) 
iris.train = iris[testidx, ]
iris.test = iris[-testidx, ]

aucs = c()
plot(x=NA, y=NA, xlim=c(0,1), ylim=c(0,1),
     ylab='True Positive Rate',
     xlab='False Positive Rate',
     bty='n')

for (type.id in 1:3) {
  type = as.factor(iris.train$Species == lvls[type.id])

  nbmodel = NaiveBayes(type ~ ., data=iris.train[, -5])
  nbprediction = predict(nbmodel, iris.test[,-5], type='raw')

  score = nbprediction$posterior[, 'TRUE']
  actual.class = iris.test$Species == lvls[type.id]

  pred = prediction(score, actual.class)
  nbperf = performance(pred, "tpr", "fpr")

  roc.x = unlist([email protected])
  roc.y = unlist([email protected])
  lines(roc.y ~ roc.x, col=type.id+1, lwd=2)

  nbauc = performance(pred, "auc")
  nbauc = unlist(slot(nbauc, "y.values"))
  aucs[type.id] = nbauc
}

lines(x=c(0,1), c(0,1))

mean(aucs)

Source of inspiration: http://karchinlab.org/fcbb2_spr14/Lectures/Machine_Learning_R.pdf