This seems like a basic questions, so I'm likely missing the big picture here...

I would like to know the AUC of each fold of a cross-validation performed in Caret's train function, in order to calculate the standard error of the AUC. Here is the typical output, which I assume provides the average "ROC" (AUC) across each of the five folds:

Generalized Linear Model 

85537 samples
   31 predictor
    2 classes: 'X0', 'X1' 

No pre-processing
Resampling: Cross-Validated (5 fold) 
Summary of sample sizes: 68430, 68429, 68430, 68430, 68429 
Resampling results:

  ROC       Sens      Spec     
  0.918912  0.834479  0.8450047

Other functions (e.g., cv.glmnet in the glmnet package) seems to output the standard error.

Call:  cv.glmnet(x = train.data.x, y = train.data.y, type.measure = c("auc"),      nfolds = 5, alpha = 1, family = "binomial") 

Measure: AUC 

       Lambda Measure        SE Nonzero
min 0.0000504  0.9027 0.0006507     166
1se 0.0003901  0.9021 0.0006224     160

Is there a statistical reason why Caret's train function does not output this as a default? I believe this would help to show the variation in the model performance for different models (e.g., logistic regression and Lasso).

I am likely operating under some false assumptions here.

  • 1
    $\begingroup$ It might just be that cv.glmnet actually uses that value to generate the "1se" model, which is the sparsest model with a performance measure within 1 standard error of the optimum. Caret doesn't seem to use the standard error for anything, so perhaps they just omit the calculation. $\endgroup$ – Nuclear Wang Mar 24 at 19:01
  • $\begingroup$ These comments might help with an explanation: "You can certainly summarize a ROC curve using a c-statistic or the AUC, but calculating confidence intervals and performing inference using c-statistics is well understood due to its relation to the Wilcoxon U-statistic. It's generally fairly well accepted that you can estimate the variability in ROC curves using the bootstrap cf Pepe Etzione Feng. This is a nice approach because the ROC curve is an empirical estimate and the bootstrap is non-parametric." $\endgroup$ – RGH Mar 24 at 19:37
  • 1
    $\begingroup$ It looks like you can examine some of that information: topepo.github.io/caret/… $\endgroup$ – Ben Reiniger Mar 24 at 19:56
  • 1
    $\begingroup$ I don't think there is a generally accepted way of estimating SE for an AUC - e.g: Hanley and McNeil (1982) provide a closed-form expression that in my opinion is often preferable to bootstrapping. Also note that every instance is exactly predicted once - Caret is probably reporting the AUC over all samples (not the mean). $\endgroup$ – Laksan Nathan Mar 24 at 20:45
  • $\begingroup$ Also found this: stat.ethz.ch/pipermail/r-help/2010-January/225327.html $\endgroup$ – RGH Mar 25 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.