Getting the bootstrap-validated AUC in R

In a paper by Faraklas et al, the researchers create a Necrotizing Soft-Tissue Infection Mortality Risk Calculator. They use logistic regression to create a model with mortality from necrotizing soft-tissue infection as the main outcome and then calculate the area under the curve (AUC). They use the bootstrap method to find the "bootstrap optimism-corrected ROC area."

If I were to do this in R, how would it look like? The code I have been toying with looks something like below:

library(boot)
library(ROCR)

auc_calc <- function(data, indices, outcomes) {
d <- data[indices,]
# Using glm for logistic regression
# Do I recreate the glm model for each dataset?
fit <- glm(outcomes[indices,] ~ X1 + X2 + X3, data=d, family=binomial)
fit.predict <- predict(fit, type="response")

# Using ROCR to calculate AUC
pred <- prediction(fit.predict, outcomes[indices,])
perf <- performance(pred, "auc")

# Returning the AUC
return(perf@y.values[[1]])
}

boot.results <- boot(data=my.data, statistic=auc_calc, R=10000, outcomes=my.outcomes)


Is this correct? Or am I doing something wrong - namely should I be passing in a glm model rather than recalculating it each time? As always thanks for the help.

• Frank Harrell's rms package has functions for this task. Fit the model with fit <- lrm(outcomes ~ X1 + X2 + X3, data=my.data, x=TRUE, y=TRUE), then use bootstrap validation with validate(fit, B=1000). The output matrix includes the optimism corrected values, but only shows Somers' $D_{xy}$. However $\text{AUC} = 0.5 \cdot D_{xy} + 0.5$. – caracal Jun 10 '13 at 17:38
• I would like to avoid using and relearning another package that would force me to rewrite what I have so far. Is there no way to do what I want using boot and ROCR? – oort Jun 11 '13 at 18:02
• Have you ever found a solution to this? – enricoferrero Aug 22 '17 at 12:26
• I emailed Harrell to ask why the validate does not output the C (auc) metric. Annoying. – CoderGuy123 Sep 21 '17 at 22:26