# Bootstrap to evaluate variance of AUC ROC

I have a toy dataset and want to eval the AUC - ROC with bootstrap

    set.seed(117)
test <- c(rnorm(70, 40, 10), rnorm(30, 55, 10))
dx <- c(rep(0, 65), rep(1,35) )
df <- data.frame(cbind(test, dx))
r <- 1000
boot.f <- function(d, i){
data <- d[i,]
p <- unlist(subset(data, data$dx==1)["test"]) n <- unlist(subset(data, data$dx==0)["test"])
mean(sample(p, r, replace=T) > sample(n, r, replace=T))
}
roc.boot<-boot(df, boot.f, r)
roc.boot
boot.ci(roc.boot, type="bca")


It seems that bootstrapping needs two samplings (in mean and in boot). Is there a more efficient way? I'd like to obtain a boot object, so prefer to use the function.

• Out of curiosity, did you get the code from this answer? If so, then just forget about it: it is plain wrong! – Calimo Feb 14 '14 at 12:30

boot is doing the bootstrapping for you. Your statistic function doesn't need to do any additional resampling - and shouldn't do it. It should only to extract the i elements that were provided, and compute the statistics on the sample.

Your problem here is that you are not computing the AUC statistic - you are (if I understand your mean line properly) sampling an estimate of it (or something of this sort, I am note quite sure how to call it here). I am unsure how bad it is to do so in bootstrap (I mean sample rather than compute), but it introduces extra variance that will (I suspect) widen your interval and that you should certainly avoid by all means.

Computing an exact AUC for your sample is not exactly trivial (i.e. it cannot be done in one line with functions from the recommended R packages), and no such function is built into R, so I would recommend using a third-party function for that. In the following I will use pROC (disclaimer: I am the developer of this package), but you could really use any function that computes the AUC.

# set.seed(117) # you should not set.seed unless you are debugging
library(pROC)
test <- c(rnorm(70, 40, 10), rnorm(30, 55, 10))
dx <- c(rep(0, 65), rep(1,35) )
df <- data.frame(cbind(test, dx))
r <- 1000
boot.f <- function(d, i){
data <- d[i,]
auc(roc(data$dx, data$test))
}
roc.boot<-boot(df, boot.f, r)
roc.boot
boot.ci(roc.boot, type="bca")


This code runs much slower than yours, partly because roc and auc perform several sanity checks, and do it at each step of the bootstrap.

It would be better to understand the literature and to know that solutions exist that do not require the bootstrap. This is just an example of a $U$-statistic, and variance estimates exist "out of the box" for $U$-statistics. See for example the implementation in the R Hmisc package rcorr.cens function. Note that Somers' $D_{xy}$ rank correlation is a simple linear translation of the $c$-index.

In addition make sure that the area of the ROC curve ($c$-index; concordance probability) is really what you need.