Often one sees, particularly in the biomedical literature, papers that analyze the performance of a risk prediction model in terms of the AUC or the area under the ROC curve. If the AUC is suitably high, then the model is taken to have "fit well" or "discriminated well". The problem is that this measure often is not validated, in that the authors do not use a separate holdout set or some sort of cross-validation procedure, even though they have more than enough data to do so.
I don't understand the reasoning behind not taking this simple step, at all. What's more, this phenomenon occurs all the time in major, reputable journals such as JAMA, BMJ, NEJM, and so on. What the heck is going on here? Have these people never heard of the concept of overfitting? Or is there something I'm missing here?