Bootstrap to evaluate variance of AUC ROC

Question

I have a toy dataset and want to eval the AUC - ROC with bootstrap

    set.seed(117)
    test <- c(rnorm(70, 40, 10), rnorm(30, 55, 10))
    dx <- c(rep(0, 65), rep(1,35) )
    df <- data.frame(cbind(test, dx)) 
    r <- 1000
    boot.f <- function(d, i){
      data <- d[i,]
      p <- unlist(subset(data, data$dx==1)["test"])
      n <- unlist(subset(data, data$dx==0)["test"])
      mean(sample(p, r, replace=T) > sample(n, r, replace=T))
    } 
    roc.boot<-boot(df, boot.f, r) 
    roc.boot 
    boot.ci(roc.boot, type="bca")

It seems that bootstrapping needs two samplings (in mean and in boot). Is there a more efficient way? I'd like to obtain a boot object, so prefer to use the function.

Answer 1

boot is doing the bootstrapping for you. Your statistic function doesn't need to do any additional resampling - and shouldn't do it. It should only to extract the i elements that were provided, and compute the statistics on the sample.

Your problem here is that you are not computing the AUC statistic - you are (if I understand your mean line properly) sampling an estimate of it (or something of this sort, I am note quite sure how to call it here). I am unsure how bad it is to do so in bootstrap (I mean sample rather than compute), but it introduces extra variance that will (I suspect) widen your interval and that you should certainly avoid by all means.

Computing an exact AUC for your sample is not exactly trivial (i.e. it cannot be done in one line with functions from the recommended R packages), and no such function is built into R, so I would recommend using a third-party function for that. In the following I will use pROC (disclaimer: I am the developer of this package), but you could really use any function that computes the AUC.

# set.seed(117) # you should not set.seed unless you are debugging
library(pROC)
test <- c(rnorm(70, 40, 10), rnorm(30, 55, 10))
dx <- c(rep(0, 65), rep(1,35) )
df <- data.frame(cbind(test, dx)) 
r <- 1000
boot.f <- function(d, i){
    data <- d[i,]
    auc(roc(data$dx, data$test))
} 
roc.boot<-boot(df, boot.f, r) 
roc.boot 
boot.ci(roc.boot, type="bca")

This code runs much slower than yours, partly because roc and auc perform several sanity checks, and do it at each step of the bootstrap.

Answer 2

It would be better to understand the literature and to know that solutions exist that do not require the bootstrap. This is just an example of a $U$-statistic, and variance estimates exist "out of the box" for $U$-statistics. See for example the implementation in the R Hmisc package rcorr.cens function. Note that Somers' $D_{xy}$ rank correlation is a simple linear translation of the $c$-index.

In addition make sure that the area of the ROC curve ($c$-index; concordance probability) is really what you need.