How to draw ROC curve with three response variable? [duplicate]

I have a toy example reproduced below in which the response variable has three possible classes. I am trying to create an ROC but not sure how to deal with it when there are three classes. Any help will be appreciated. Thanks

library(ipred)
control = rpart.control(maxdepth = 20, minsplit = 20, cp = 0.01, maxsurrogate=2, surrogatestyle = 0, xval=25)
n <- 500; p <- 10
f <- function(x,a,b,d) return( a*(x-b)^2+d )
x1 <- runif(n/2,0,4)
y1 <- f(x1,-1,2,1.7)+runif(n/2,-1,1)
x2 <- runif(n/2,2,6)
y2 <- f(x2,1,4,-1.7)+runif(n/2,-1,1)
y <- c(rep(-1,floor(n/3)),rep(0,ceiling(n/3)), rep(1,ceiling(n/3)))
dat <- data.frame(y=factor(y),x1=c(x1,x2),x2=c(y1,y2), matrix(rnorm(n*(p-2)),ncol=(p-2)))
names(dat)<-c("y",paste("x",1:p,sep=""))
dat

plot(dat$x1,dat$x2,pch=c(1:2)[y], col=c(1,8)[y],
xlab=names(dat)[2],ylab=names(dat)[3])
indtrain<-sample(1:n,300,replace=FALSE)
train<-dat[indtrain,]; dim(train)
test<-dat[setdiff(1:n,indtrain),]; dim(test)
test

mod <- bagging(y~.,  data=train, control=control, coob=TRUE, nbagg=25, keepX = TRUE)
mod
pred<-predict(mod, newdata=test[,-1],type="prob", aggregation= "average"); pred


For two class case, I use to do the following but it is no longer valid for three classes.

yhat <- pred[,2]
y = test[, -1]
plot.roc(y, yhat)

• You are right -- the ROC is defined only for two classes, and perhaps no obvious generalization exists. One option is break your analysis down in to three separate ROCs (A versus B union C, B versus A union C, and C versus A union B). Alternatively, the overall misclassification rate could be of interest. Commented Oct 2, 2013 at 15:30
• Thanks @zkurtz, As you suggested how can I find union using class probabilities? Commented Oct 2, 2013 at 15:39

You might want to have a look at the Volume Under the ROC Surface as defined in the following articles:

ROC Analysis was designed for dealing with only two variables: noise and no noise, so using it for 3 or more variables makes little sense.

However, you for any multi-classification problem it's possible to use a bunch of binary classifiers and do so-called One-Vs-All Classification

E.g. consider the IRIS data set: there are 3 classes: setosa, versicolor, and virginica. So we can build 3 classifiers (e.g. Naive Bayes): for setosa, for vesicolor and for virginica. And then draw a ROC curve for each and tune the threshold for each model separately. AUC in such a case could be just the average across AUCs for individual models.

Here's a ROC curve for the IRIS data set:

AUC in this case is $\approx 0.98 = \frac{1 + 0.98 + 0.97}{3}$

R Code:

library(ROCR)
library(klaR)

data(iris)

lvls = levels(iris$Species) testidx = which(1:length(iris[, 1]) %% 5 == 0) iris.train = iris[testidx, ] iris.test = iris[-testidx, ] aucs = c() plot(x=NA, y=NA, xlim=c(0,1), ylim=c(0,1), ylab='True Positive Rate', xlab='False Positive Rate', bty='n') for (type.id in 1:3) { type = as.factor(iris.train$Species == lvls[type.id])

nbmodel = NaiveBayes(type ~ ., data=iris.train[, -5])
nbprediction = predict(nbmodel, iris.test[,-5], type='raw')

score = nbprediction$posterior[, 'TRUE'] actual.class = iris.test$Species == lvls[type.id]

pred = prediction(score, actual.class)
nbperf = performance(pred, "tpr", "fpr")

roc.x = unlist([email protected])
roc.y = unlist([email protected])
lines(roc.y ~ roc.x, col=type.id+1, lwd=2)

nbauc = performance(pred, "auc")
nbauc = unlist(slot(nbauc, "y.values"))
aucs[type.id] = nbauc
}

lines(x=c(0,1), c(0,1))

mean(aucs)


Source of inspiration: http://karchinlab.org/fcbb2_spr14/Lectures/Machine_Learning_R.pdf

• Should the thresholds be set separately for each class? If yes, What if 2 classes have the same threshold? Commented Jun 17, 2019 at 10:23
• @Alexey how can we calculate the confidence interval in this example? Commented Jun 6, 2022 at 23:53