I have a pretty small data set with (approx 150 obs) that I'm using to predict both a binary outcome variable and a continuous. Right now, I'm using nested cross validation as follows:
Outer loop is 20 times repeated 5-fold CV to produce 100 performance metrics. Each inner loop is 10-fold CV to tune hyper parameters.
I train a number of different algorithms (OLS, Lasso, Random Forest) for both problems yielding 100 measures of AUC and correlation coefficients for the classifier and regression respectively.
I've been trying to figure out what would be the best way to produce confidence intervals for these metrics and to do statistical tests that the predictive accuracy is better than random, and I'm sort of lost (samples are not independent). One option that has been presented here is to use a bootstrap (e.g. Confidence intervals for cross-validated statistics):
I'm not sure if I understood the procedure. Would this be the correct way to do it?
- Instead of the outer CV loop do a bootstrap of the inner CV, where I train the models on a training set that consists of 80% of a sample with n=150 drawn with replacement.
- In each bootstrap iteration, I then run a 10-fold CV to grid-search for hyper parameters.
- After finding the best parametrization using the inner CV, I test it on the 20% remaining data, producing one performance metric.
- Repeat 1000 to get enough measures.
- Confidence Interval is then just between 2.5 and 97.5 percentile of observed metrics.
Problem is that this method would take ages to run. I also found this R package that could give CI for the AUC without me having to do 1000 iterations: https://github.com/ledell/cvAUC. I could then get a CI and p-value for rho just by running the outer CV 5 times instead and collecting all the predictions (150) to produce one rho value with its regular CI and p-value (H0: >0). Should I do this instead of the bootstrap, or how do the two alternatives compare?