Nested cross validation (NCV) is the standard procedure to estimate the performance of a classifier, after tuning its parameters and hyper-parameters. Despite being a concept quite general and widely known, there seem to me to be no clear receipt on how to compute the error on this performance estimate. Surely, the question has been asked elsewhere on 'Cross Validated', for example here, or here, or here. However none of those answers or questions is highly voted, as I would expect for a question so general and important. Nor can I find an answer to my questions elsewhere on the internet.
In sum, I found the following answers, and I cannot decide myself for one:
- use wilson score interval, see here. In this case the error band only depends on the performance itself (which sounds strange to me. furthermore it only applies to accuracy, for what I understood, not to other performance measures as AUC for ROC)
- since the NCV estimate is the mean on the N outer folds, compute the SE of this mean and Wald intervals (this is what I would have done before reading around). If this answer is correct, why would one need bootstrap and/or repeated CV, see below?
- same as 2. but with repeated cross validation, so one has more measures (which however are correlated then), the issues inherent to this choice are explained here. Nice insightful paper, that offers no solution though.
- bootstrapping, but computationally veeery expensive as NCV is per se already expensive.
- permutations (practically too expensive, see above)
Therefore, what would be the recommended procedure, say in a problem with 1000 participants and 100 features and a grid search over say 100 hyperparameters combinations? How would that change for 10k participants and 1k features?