How to compute confidence interval for leave-one-out cross-validated AUC that is also repeated many times?

Question

I have a small dataset of 100 data points and trained a random forest classifier using nested leave-one-out cross-validation. The details go like this:

• In each trial out of 10:

  • for each leave-one-out patient out of 100:

    • take the remaining 99 patients, do 10-fold cross-validation to optimize for the best hyperparameters using grid search. Take the best performing set of parameters (based on 10-fold cv average), and evaluate on the leave-one-out patient.

Because inner 10-fold cv is random, the models trained in each trial are different (have different parameters) and as a result, if I do 10 trials, I get 10 x 100 models and 10 x 100 predictions. I can calculate 10 ROC curves for the entire set of 100 patients and calculate the AUC confidence interval for each of 10 curves using cvAUC.

My question is, would it make sense to consider the 10 trials as simply additional leave-one-out validation splits? In other words, what is the statistical consequence if I simply pool the 10 x 100 predictions and treat it like leave-one-out cross-validation on 1000 patients and derive its confidence interval? and is there a better way to do this?

Thanks!

Answer 1

I’d remove the in each trial bit in the first line, that doesn’t make sense to me... With nested cross validation you have one nested for loop, not two.

Then you have for each leave one out do inner cv on the 99 observations to get parameters and then fit the model on the 99, then you have one output prediction per observation in nested LOOCV. So there is one ROC and one AUC for the 100 cross validated probabilities.

You can get the CI of the AUC through an equation if required. https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Confidence_Intervals_for_the_Area_Under_an_ROC_Curve.pdf