How to combine two uncertainties in estimation of (binary) classification metrics?

Question

I have a data set with 22990 samples. I am training a binary classification model (logistic regression) and use 67%/33% splitting for train-validate/test sets. The model is 10-fold cross validated. To assess the stability of the metrics on the test set due to random splitting, I repeat the train-validate/test procedure 100 times.

QUESTION

How to properly report results? I have two sources for errors of metrics - one is the binomial error estimation and the second is the variability due to 100 repetition.

If I want to estimate the binomial interval, how do I do it as I have 100 repetitions?

Answer 1

Logistic regression is not a classification method. And you don't need two levels of data splitting for this problem. You can use the Efron-Gong "optimism" bootstrap (see for example the R rms package validate.lrm function) for the validation.

Logistic regression is a probability estimation model, and indexes of predictive ability for it should be chosen from among the class of proper probability accuracy scores and the $c$-index (concordance probability; area under the ROC curve). For more details see http://www.fharrell.com/2017/03/damage-caused-by-classification.html and http://www.fharrell.com/2017/01/classification-vs-prediction.html .