I have a toy example reproduced below in which the response variable has three possible classes. I am trying to create an ROC but not sure how to deal with it when there are three classes. Any help will be appreciated. Thanks

control = rpart.control(maxdepth = 20, minsplit = 20, cp = 0.01, maxsurrogate=2, surrogatestyle = 0, xval=25)
n <- 500; p <- 10
f <- function(x,a,b,d) return( a*(x-b)^2+d )
x1 <- runif(n/2,0,4)
y1 <- f(x1,-1,2,1.7)+runif(n/2,-1,1)
x2 <- runif(n/2,2,6)
y2 <- f(x2,1,4,-1.7)+runif(n/2,-1,1)
y <- c(rep(-1,floor(n/3)),rep(0,ceiling(n/3)), rep(1,ceiling(n/3)))
dat <- data.frame(y=factor(y),x1=c(x1,x2),x2=c(y1,y2), matrix(rnorm(n*(p-2)),ncol=(p-2)))

plot(dat$x1,dat$x2,pch=c(1:2)[y], col=c(1,8)[y], 
train<-dat[indtrain,]; dim(train) 
test<-dat[setdiff(1:n,indtrain),]; dim(test) 

mod <- bagging(y~.,  data=train, control=control, coob=TRUE, nbagg=25, keepX = TRUE)
pred<-predict(mod, newdata=test[,-1],type="prob", aggregation= "average"); pred

For two class case, I use to do the following but it is no longer valid for three classes.

yhat <- pred[,2]
y = test[, -1]
plot.roc(y, yhat)
  • $\begingroup$ You are right -- the ROC is defined only for two classes, and perhaps no obvious generalization exists. One option is break your analysis down in to three separate ROCs (A versus B union C, B versus A union C, and C versus A union B). Alternatively, the overall misclassification rate could be of interest. $\endgroup$
    – zkurtz
    Oct 2, 2013 at 15:30
  • $\begingroup$ Thanks @zkurtz, As you suggested how can I find union using class probabilities? $\endgroup$ Oct 2, 2013 at 15:39

2 Answers 2


You might want to have a look at the Volume Under the ROC Surface as defined in the following articles:


ROC Analysis was designed for dealing with only two variables: noise and no noise, so using it for 3 or more variables makes little sense.

However, you for any multi-classification problem it's possible to use a bunch of binary classifiers and do so-called One-Vs-All Classification

E.g. consider the IRIS data set: there are 3 classes: setosa, versicolor, and virginica. So we can build 3 classifiers (e.g. Naive Bayes): for setosa, for vesicolor and for virginica. And then draw a ROC curve for each and tune the threshold for each model separately. AUC in such a case could be just the average across AUCs for individual models.

Here's a ROC curve for the IRIS data set:

ROC Curve

AUC in this case is $\approx 0.98 = \frac{1 + 0.98 + 0.97}{3}$

R Code:



lvls = levels(iris$Species)
testidx = which(1:length(iris[, 1]) %% 5 == 0) 
iris.train = iris[testidx, ]
iris.test = iris[-testidx, ]

aucs = c()
plot(x=NA, y=NA, xlim=c(0,1), ylim=c(0,1),
     ylab='True Positive Rate',
     xlab='False Positive Rate',

for (type.id in 1:3) {
  type = as.factor(iris.train$Species == lvls[type.id])

  nbmodel = NaiveBayes(type ~ ., data=iris.train[, -5])
  nbprediction = predict(nbmodel, iris.test[,-5], type='raw')

  score = nbprediction$posterior[, 'TRUE']
  actual.class = iris.test$Species == lvls[type.id]

  pred = prediction(score, actual.class)
  nbperf = performance(pred, "tpr", "fpr")

  roc.x = unlist(nbperf@x.values)
  roc.y = unlist(nbperf@y.values)
  lines(roc.y ~ roc.x, col=type.id+1, lwd=2)

  nbauc = performance(pred, "auc")
  nbauc = unlist(slot(nbauc, "y.values"))
  aucs[type.id] = nbauc

lines(x=c(0,1), c(0,1))


Source of inspiration: http://karchinlab.org/fcbb2_spr14/Lectures/Machine_Learning_R.pdf

  • $\begingroup$ Should the thresholds be set separately for each class? If yes, What if 2 classes have the same threshold? $\endgroup$
    – 3nomis
    Jun 17, 2019 at 10:23
  • $\begingroup$ @Alexey how can we calculate the confidence interval in this example? $\endgroup$
    – Melih Aras
    Jun 6 at 23:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.