# R: Caret package: AUC (“ROC”) standard error from cross validation

This seems like a basic questions, so I'm likely missing the big picture here...

I would like to know the AUC of each fold of a cross-validation performed in Caret's train function, in order to calculate the standard error of the AUC. Here is the typical output, which I assume provides the average "ROC" (AUC) across each of the five folds:

Generalized Linear Model

85537 samples
31 predictor
2 classes: 'X0', 'X1'

No pre-processing
Resampling: Cross-Validated (5 fold)
Summary of sample sizes: 68430, 68429, 68430, 68430, 68429
Resampling results:

ROC       Sens      Spec
0.918912  0.834479  0.8450047


Other functions (e.g., cv.glmnet in the glmnet package) seems to output the standard error.

Call:  cv.glmnet(x = train.data.x, y = train.data.y, type.measure = c("auc"),      nfolds = 5, alpha = 1, family = "binomial")

Measure: AUC

Lambda Measure        SE Nonzero
min 0.0000504  0.9027 0.0006507     166
1se 0.0003901  0.9021 0.0006224     160


Is there a statistical reason why Caret's train function does not output this as a default? I believe this would help to show the variation in the model performance for different models (e.g., logistic regression and Lasso).

I am likely operating under some false assumptions here.

• It might just be that cv.glmnet actually uses that value to generate the "1se" model, which is the sparsest model with a performance measure within 1 standard error of the optimum. Caret doesn't seem to use the standard error for anything, so perhaps they just omit the calculation. – Nuclear Wang Mar 24 at 19:01
• These comments might help with an explanation: "You can certainly summarize a ROC curve using a c-statistic or the AUC, but calculating confidence intervals and performing inference using c-statistics is well understood due to its relation to the Wilcoxon U-statistic. It's generally fairly well accepted that you can estimate the variability in ROC curves using the bootstrap cf Pepe Etzione Feng. This is a nice approach because the ROC curve is an empirical estimate and the bootstrap is non-parametric." – RGH Mar 24 at 19:37
• It looks like you can examine some of that information: topepo.github.io/caret/… – Ben Reiniger Mar 24 at 19:56
• I don't think there is a generally accepted way of estimating SE for an AUC - e.g: Hanley and McNeil (1982) provide a closed-form expression that in my opinion is often preferable to bootstrapping. Also note that every instance is exactly predicted once - Caret is probably reporting the AUC over all samples (not the mean). – Laksan Nathan Mar 24 at 20:45
• Also found this: stat.ethz.ch/pipermail/r-help/2010-January/225327.html – RGH Mar 25 at 14:18