Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

Question

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking:

Please provide statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

I found some answers: basically I split my cohort into train set (n=107) and test set (n=12), did 2 ROC curves and compared AUC for both using this command:

testobj <- roc.test(rocobj0, rocobj1);testobj 

and this was the output

DeLong's test for two ROC curves

data: rocobj0 and rocobj1
D = 0.44822, df = 12.488, p-value = 0.6617
alternative hypothesis: true difference in AUC is not equal to 0
sample estimates:
AUC of roc1 AUC of roc2
80.26961 72.22222

Regarding the optimism adjustment, I did it using the following code copied from a blog post on cainarchaeology.

library (pROC);library (kimisc) 
  auc.adjust <- function(data, fit, B){
  fit.model <- fit
  data$pred.prob <- fitted(fit.model)
  auc.app <- roc(data\[,1\], data$pred.prob, data=data)$auc # require 'pROC'
  auc.boot <- vector (mode = "numeric", length = B)
  auc.orig <- vector (mode = "numeric", length = B)
  o <- vector (mode = "numeric", length = B)
  for(i in 1:B){    
    boot.sample <- sample.rows(data, nrow(data), replace=TRUE) # require 'kimisc'
    fit.boot <- glm(formula(fit.model), data = boot.sample, family = "binomial")
    boot.sample$pred.prob <- fitted(fit.boot)
    auc.boot\[i\] <- roc(boot.sample\[,1\], boot.sample$pred.prob, data=boot.sample)$auc
    data$pred.prob.back <- predict.glm(fit.boot, newdata=data, type="response")
    auc.orig\[i\] <- roc(data\[,1\], data$pred.prob.back, data=data)$auc
    o\[i\] <- auc.boot\[i\] - auc.orig\[i\]
  }
  auc.adj <- auc.app - (sum(o)/B)
  boxplot(auc.boot, auc.orig, names=c("auc.boot", "auc.orig"))
  title(main=paste("Optimism-adjusted AUC", "\nn of bootstrap resamples:", B), sub=paste("auc.app (blue line)=", round(auc.app, digits=4),"\nadj.auc (red line)=", round(auc.adj, digits=4)), cex.sub=0.8)
  abline(h=auc.app, col="blue", lty=2)
  abline(h=auc.adj, col="red", lty=3)
}

model <- glm(data$y ~  data$x, data = data, family = "binomial")
auc.adjust(data, model, B=200)

But it gave me this weird figure below in which AUC is > 1:

1. I should conclude that AUC was not overfitted based on the splitting of the data that I did and P value of 0.66 from DeLong's test for two ROC curves, Right?
2. How to fix the optimism adjusted AUC figure to get it <=1?

Answer 1

A single test/train breakdown, your first suggestion, is not going to work well for evaluating overfitting in your situation with only 119 cases. Separating out one test and one training set doesn't usually work well unless there are thousands of cases, as otherwise the results depend too heavily on the vagaries both of the sample from the population and the particular choice of test set out of your sample.

The second approach, based on bootstrapping, is well accepted and works well with limited data sets like yours. For each bootstrap sample you develop a model and compare the AUC on the bootstrap sample against the AUC from applying that model to the original data set. The difference is a measure of the optimism of the model that was developed on the bootstrap sample. The idea is that bootstrap sampling from your data set is like taking the original data set from the full population, so this optimism (averaged over multiple bootstrap samples) estimates the optimism in your original model.

It's not clear why your code produced such strange looking results, and coding questions per se are not on topic here. (I do note that in one place you call your data set data and in other places df.) You can avoid coding problems by using well documented statistical functions. The rms package in R, for example, provides an lrm() function to do logistic regression and a validate() function that evaluates optimism (of several measures) directly on lrm output and provides optimism-adjusted values. It might not show AUC directly, but you can get that from the $D_{xy}$ value it reports: AUC = $0.5 + (D_{xy}/2)$. The rms package is from Frank Harrell, acknowledged in the blog post you cite as the originator of this approach to evaluating and correcting for optimism.

Answer 2

Cross-validation is another approach to avoid over-optimism. I ran a logistic model with your data, using 4-fold cross-validation (i.e. fit the model on 75% of the cases, and then measure the error on the remaining 25% -- then do this 3 more times).

I get a cross-validated ROC area of .788, so your .81 looks pretty accurate. I think EdM's suggestion of using Frank Harrell's work is going to work better with the reviewers, though.