8
$\begingroup$

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net.

The procedure I use is as follows:

1) build model using the entire dataset, obtain predicted values, and calculate AUC (AUC_ap, apparent)

2) generate 100-500 bootstrap samples derived from the original dataset

3) for each bootstrap sample, follow the identical procedure as in #1, and obtain predicted values and auc for i) current bootstrap sample, and ii) original dataset

4) calculate difference between i) and ii) (in #3) for each of the 100-500 bootstrap sample, and take the average --> "optimism"

5) calculate optimism-corrected AUC : AUC_ap - optimism

My question is WHAT ROC curve would be best to present in a paper? For example, the ROC derived in step #1 is one choice, but clearly optimistic. Alternatively, I have tried to generate an "average ROC" using the R package ROCR, based on the ROC curves derived in step #3 (ii). However, the AUC for the [average of these ROC curves] I do not believe is equivalent to the value obtained in step #5.

Any input is greatly appreciated! -M

Improve this question
$\endgroup$

3 Answers 3

Reset to default
6
$\begingroup$

You are making the assumption that the ROC curve is informative and leads to good decisions. Neither is true. I have yet to see an ROC curve that provided useful insight. It also has a large ink:information ratio. The $c$-index (concordance probability) is a good measure of predictive discrimination. I would like it better were it not also the AUROC. There is no need to present an ROC curve.

Besides having low information yield, ROC curves invite analysts to seek cutpoints on predicted probabilities, which is a decision-making disaster.

Improve this answer
$\endgroup$
0
$\begingroup$

You raised a very good question that I was wondering for a long time. Perhaps it depends on your results to make a decision how to report. For most situations, authors would like to report a raw/apparent AUC (ie. step #1 in your question) despite over-optimism or not, and then report the bootstrap optimism corrected AUC (ie. step #5). see ref: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0125026

In some situations that AUC seems not to be over-optimistic, author would directly report corrected AUC.

As for AUC in step #3 (ii), it has rarely been reported and you'd better ignore it.

Improve this answer
$\endgroup$
-2
$\begingroup$

There are many details missing your question - however it appears to me that you are not talking about test set all. If you intend to demonstrate the generalizability of your model (which is primary use case for an ROC curve), you are expected to present the ROC derived from a test set, not validation or internal validation set. or an average ROC derived from multiple test sets. Hence it is important you find a way to generate test sets, and take it from there.

A good reference to learn ROC analysis (and how to create average ROC curves) is:

Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. http://www.sciencedirect.com/science/article/pii/S016786550500303X

Improve this answer
$\endgroup$
1
  • $\begingroup$ Creating a test set from the same data stream is still internal validation and is less reliable than using the optimism bootstrap. Split-sample validation is incredibly inefficient and indeed often misleading. I discuss this in detail in Biostatistics for Biomedical Research Section 10.11 available from biostat.mc.vanderbilt.edu/ClinStat $\endgroup$
    – Frank Harrell
    Commented Mar 6, 2016 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.