# Calculate a 95% confidence interval and p-value for the change in C-statistic using bootstrap with R

I am using boot() and boot.ci() to furnish confidence intervals for the difference in the $c$-statistic (AUC) between models with and without a novel biomarker. My method is below. I am not sure if this is the correct way to go about it.

Here is example data (n = 500, 3 columns: disease status (1/0), covariate (1/0) and biomarker (1-6)):

data <- read.table("https://notendur.hi.is/eyb8/dataframe.txt", header=TRUE)


Here is a function that returns the difference in $c$-statistic between the models, for use in boot():

library(rms)
Cindexdiff <- function(data, indices){
data <- data[indices,] # select obs. in bootstrap sample
# C-statistic without the biomarker:
C1   <- lrm(disease ~ covariate, data=data, x=TRUE, y=TRUE)$stats["C"] # C-statistic with the biomarker: C2 <- lrm(disease ~ biomarker + covariate, data=data, x=TRUE, y=TRUE)$stats["C"]
as.numeric(C2-C1) # returns the difference
}


And now the 95% CI for the difference in $c$-statistics with bootstrapping (repeated 999 times):

library(boot)
set.seed(1)
b <- boot(data, Cindexdiff, 999)
boot.ci(b)


Now, I have a few questions:

1. Is this a sound method for calculating bootstrap confidence interval for the change in C-statistic?

2. How should I choose which type of confidence interval from boot.ci() to use? "Normal" or bias-corrected (BCa)?

• There is good discussion below on why one might not want to do hypothesis testing with the C-index. However, I still would like to know whether my script is correct. Is this the way to bootstrap in R? Commented Jun 25, 2014 at 22:04

The $c$-index is not a good basis for a statistical comparison of competing models; it lacks power for that. You can use the gold standard likelihood ratio $\chi^2$ test or use a special $c$-statistic for paired prediction assessment (R Hmisc package rcorrp.cens function) that has much more power than the difference of $c$'s. rcorrp.cens asks the question "in random pairs of observations with paired predictions from two models with equal amounts of overfitting is one more more concordant than the other?". But there is no reason not to use the age-old partial effect test of association.
• The $c$ statistic's precision makes it not a good choice for comparison in that way. Commented Jun 24, 2014 at 14:40
• Don't perform an analysis that we know is suboptimal. The fact that it has been used countless times in the literature does not predict the probability that it is a good idea. As the person who developed the $c$-index I've had a fair amount of experience in this area. Think of $\Delta c$ as comparing two Wilcoxon statistics for A vs. B and A vs. C to compare B vs. C. No one would do that. And the inventor of IDI and NRI recently stated he would never use them for statistical tests but would instead use the likelihood ratio $\chi^2$ statistic. Commented Jun 24, 2014 at 19:35
• I would use things that are derived from likelihood or deviance such as generalized $R^2$ along with plotting predicted values for big model vs. predicted values for smaller model. Commented Jun 24, 2014 at 22:57