What's the reasoning behind presenting unvalidated AUC as a measure of model fit or performance?

Question

Often one sees, particularly in the biomedical literature, papers that analyze the performance of a risk prediction model in terms of the AUC or the area under the ROC curve. If the AUC is suitably high, then the model is taken to have "fit well" or "discriminated well". The problem is that this measure often is not validated, in that the authors do not use a separate holdout set or some sort of cross-validation procedure, even though they have more than enough data to do so.

I don't understand the reasoning behind not taking this simple step, at all. What's more, this phenomenon occurs all the time in major, reputable journals such as JAMA, BMJ, NEJM, and so on. What the heck is going on here? Have these people never heard of the concept of overfitting? Or is there something I'm missing here?

Answer 1

This is getting less prevalent but as you said still occurs. It should not be allowed by journal editors or reviewers. My preference is for rigorous bootstrap overfitting-corrected measures of model performance (Efron-Gong optimism bootstrap). Note that the $c$-index (concordance probability; ROC area) has little to do with model goodness of fit but is a measure of pure discrimination. I want to see Brier score and generalized $R^2$ measures in addition to $c$. I really want to see nonparametric or semi-parametric calibration curves, de-biased by bootstrapping.