Is it valid to do ROC analysis without using test data? I am trying to do an ROC analysis to check whether a particular biomarker has any diagnostic power for screening a condition. Since I have only small dataset of 100. I am not splitting the data into training and testing. Is it valid to report the AUC by just doing the analysis on training data set?
 A: This is exactly the kind of situation Harrell means when he says not to do splitting at all.
His recommendation is to fit the model to the entire dataset and measure performance. To check for overfitting, use a bootstrap approach.

*

*Draw a sample, with replacement, of feature-y pairs from your dataset. This is a bootstrap sample.


*Fit your model to the bootstrap sample.


*Apply the bootstrap model to the entire dataset.


*Compare the performance to the performance of the model that was trained on the whole dataset. Performance would be measured by something like log loss or Brier score.


*Repeat this dozens or hundreds of times to get a distribution of comparisons.
If the distribution resulting from this process indicates that the model trained on all of the data is not much better than the bootstrap models applied to the whole dataset, then the model does not suffer from overfitting.
The R function rms::validate does this for you.
References
Regression Modeling Strategies by Frank Harrell
“Why R” Keynote by Frank Harrell
