1
$\begingroup$

I have developed an SVM-Model using x data. ROC curve was generated using 5-fold cross-validation. Now I want to compare my new SVM-model with a published Bayes-classifier. I have got the predictions for x data, using both published model and the SVM-model.

Data looks like:

True-Label, Predicted labels

Item True-Label Model-1 Model-2
 AFA  1          0      1
 AFB  0          0      0
 AFD  0          1      1
 BFA  1          1      1
 .....
 .....
 .....
 ZFD  0          1       0

1) Can an ROC Curve be generated using above data, if yes, how to generate one? Will it be appropriate to compare these curves with each other?

2) What are the other approaches to compare these kind of models(I have calculated metrics like sensitivity, specificity,...).

EDIT-1

For the published model I have TPR and FPR values (not for same data). Bayes classifier also outputs probability score along with label. How can I use it for generating Curves?

EDIT-2
Data generated from prediction by Bayes Classifier Contains Probability Values. I Formatted the output into following table (for x data).

True-Label Predicted-Label Probability
0          0               0.27
0          1               0.97
1          1               0.6
1          1               0.978

Can an ROC curve be generated using this data ?

$\endgroup$
1
$\begingroup$

As Marc Claesen points out, some kind of certainty measure is needed. Below I have showed two approaches of how to form ROC curves.

  1. If the classifier can output a probabilistic measure, such one can be used in e.g. 5-fold cross validation to form a ROC plot.
  2. If the classifier only outputs predicted labels, then the certainty of predictions can be estimated with bagging. The training set is bootstrapped and modeled e.g. 100 times and the cross validated out-of-bag predictions are used for ROC curves.

(1,probabilistic svm, black curve) and (2,bagged svm, red curve) enter image description here

For multi-class ROC curves use e.g. "1 vs. rest" method, check out this post

rm(list=ls())
set.seed(1)
library(e1071)
library(AUC)

data(iris)
iris = iris[1:100,] #remove one species, to simplify to a 2-class problem
iris[1:4] = lapply(iris[1:4],jitter,amount=2) #add noise, otherwise too easy
#NB ROC PLOT will change for each new random noise component (jitter)
X = iris[1:100,names(iris)!="Species"]
y = iris[1:100,"Species"]

#cross-validated SVM-probability plot
folds = 5
test.fold = split(sample(1:length(y)),1:folds) #ignore warning
all.pred.tables = lapply(1:folds,function(i) {
  test = test.fold[[i]]
  Xtrain = X[-test,]
  ytrain = y[-test ]
  sm = svm(Xtrain,ytrain,prob=T) #some tuning may be needed
  prob.benign = attr(predict(sm,X[test,],prob=T),"probabilities")[,2]
  data.frame(ytest=y[test],ypred=prob.benign) #returning this
})
full.pred.table = do.call(rbind,all.pred.tables)
plot(roc(full.pred.table[,2],full.pred.table[,1]))


#bagged OOB-cross validated SVM AUC plot
n.bootstraps=100 #how many models to train
inbag.matrix = replicate(n.bootstraps,sample(1:length(y),replace=T))
all.preds = sapply(1:n.bootstraps,function(i) {
  inbag = inbag.matrix[,i]
  outOfBag = which(!1:length(y) %in% inbag)
  Xtrain = X[inbag,]
  ytrain = y[inbag ]
  sm = svm(Xtrain,ytrain) #some tuning may be needed
  pred.label = rep(NA,length(y))
  pred = predict(sm,X[outOfBag,])
  pred.label[outOfBag] = levels(pred)[as.numeric(pred)]
  addNA(factor(pred.label))
})

bag.prob = apply(all.preds,1,function(aRow){
  inbag = which(is.na(aRow))
  mean(aRow[-inbag] == levels(y)[2])
})
plot(roc(bag.prob,y),col="red",add=TRUE)
$\endgroup$
2
$\begingroup$

Unfortunately, you can't generate ROC curves from that data. To construct ROC curves, you have to be able to rank the test set from strong positive to strong negative. For that you need scores, which measure the level of confidence that a data instance belong to the positive class (e.g. signed distance to the separating hyperplane for SVM and predicted probability for Naive Bayes).

Predicted labels alone do not contain enough information to construct ROC curves.

In response to edit-2: yes, you can construct an ROC curve based on that data.

$\endgroup$
  • $\begingroup$ To construct an ROC curve you would need TPR and FPR pairs at each rank, not overall (overall is based on some arbitrary choice of threshold on the scores). The data must be the same for any reasonable comparison. $\endgroup$ – Marc Claesen Oct 8 '15 at 6:00
  • $\begingroup$ Can you briefly explain the steps involved to draw the curve. It will be very helpful. $\endgroup$ – VAR121 Oct 8 '15 at 17:10
  • $\begingroup$ @VAR121 I have posted two approaches to draw ROC curves $\endgroup$ – Soren Havelund Welling Oct 12 '15 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.