The different usage of ROC between diagnotic tests and machine learning

Question

I am currently very confused about the ROC usage in diagnostic tests and machine learning. In the scenario of medical diagnostic studies, many tutorial does not mention the data split procedure as often highlighted in machine learning courses, and generally draw the ROC with all the available data to assess the diagnotic performance of a specific clinical test. However, when the data analysts attempts to assess the model performance of several models, the analysts often perform the data split/cross validation and the ROC/AUC is often only applied for the test data only.

What is the difference between the two scenarios and why the researches performed the ROC/AUC analysis in different ways?

Hope anyone can help me with this question. Thanks in advance!

Answer 1

The difference between the two scenarios is usually bad practice an it is not specific to AUC-ROC. We should always have a validation procedure; either by repeated resampling and/or an adequate hold-out set irrespective of our performance metric. In fairness, medical practice (and presentation of data analysis) has moved ahead from that but yes, especially pre-2010s papers report in-sample estimates for performance metrics without realising these estimates are often biased (in a way that benefits the algorithm used). This is why we had publication like Why Most Published Research Findings Are False (2005) by Ioannidis honing in exactly at that problem and we had the replication crisis discussions throughout the last decade.