# Differences between cross validation and bootstrapping to estimate the standard error of the AUC of a given ROC curve

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier.

For example, in here (Differences between cross validation and bootstrapping to estimate the prediction error), here (Bootstrapping estimates of out-of-sample error) and here (What is the .632+ rule in bootstrapping?).

I'm interested in, however, maximizing AUC directly, and not the prediction error (1 - accuracy) itself as the cutoff points are not specified a priori.

Would the reasoning of the posts above apply? I find it difficult, for example, to calculate the AUC with only 10 observations (assuming a CV10 applied to a sample of 100 observations).

Currently I'm using the "optimism" bootstrap estimator, though it is pretty expensive (at least for the PC I access to).

Any thoughts?

• See this page for further information, including some thoughts on why AUC might not be the best evaluation of your model. – EdM Jul 29 '15 at 19:18
• AUC seems to be the more "neutral" evaluation metric as I don't have to specify a threshold (which is, in itself, a different problem altogether), required to calculate accuracy, for example. – Leonardo Cordeiro Aug 11 '15 at 19:31

• Which flavor of bootstrap estimator do you use? Elsewhere Frank Harrell advocated the "optimism" boostrap but I can't seem to find a proper reference. Thoughts? I'm aware of what you mean about the AUC. I can't estimate error costs for this particular problem so I'm good with the whole AUC. AIC and BIC don't apply to my problem, I reckon. The number of parameters is fixed and theres no likelihood function. – Leonardo Cordeiro Aug 16 '15 at 1:21
• Try looking at the functions provided in Harrell's rms` package in R, and at the notes on his academic website. – EdM Aug 16 '15 at 3:12