I have a small dataset of 100 data points and trained a random forest classifier using nested leave-one-out cross-validation. The details go like this:
In each trial out of 10:
for each leave-one-out patient out of 100:
- take the remaining 99 patients, do 10-fold cross-validation to optimize for the best hyperparameters using grid search. Take the best performing set of parameters (based on 10-fold cv average), and evaluate on the leave-one-out patient.
Because inner 10-fold cv is random, the models trained in each trial are different (have different parameters) and as a result, if I do 10 trials, I get 10 x 100 models and 10 x 100 predictions. I can calculate 10 ROC curves for the entire set of 100 patients and calculate the AUC confidence interval for each of 10 curves using cvAUC.
My question is, would it make sense to consider the 10 trials as simply additional leave-one-out validation splits? In other words, what is the statistical consequence if I simply pool the 10 x 100 predictions and treat it like leave-one-out cross-validation on 1000 patients and derive its confidence interval? and is there a better way to do this?