Benefit of nested CV for algorithm comparison? Let's consider some small classification dataset D, and 2 classifiers A and B:

*

*a classifier A, with hyper-parameter $\alpha$, and a set of candidate values for $\alpha$: (1, 10)

*a classifier B, with hyper-parameter $\beta$, and some set of candidate values for $\beta$: (5, 500)

The goal is to identify the best performing classifier on D: A or B? Which hyper-parameter value? Which parameter value?
I can think of 2 approaches to answer this question:

*

*Without nested CV: estimate the k-fold CV error for each value of $\alpha$ and each value of $\beta$. For example, we could get: ($\alpha$=1, err=0.7), ($\alpha$=10, err=0.1), ($\beta$=5, err=0.6), ($\beta$=500, err=0.9). The lowest error is achieved by (A, $\alpha$=10). We retrain A with $\alpha$=10 on the full dataset, and that's our final model.

*With nested CV: estimate the test errors of A and B using nested CV. For example, we could get (A, err=0.3), (B, err=0.5). We now know that A is better, but we don't know which value to pick for $\alpha$. So we do a regular k-fold CV for model selection. For example, we could get ($\alpha$=1, err=0.7), ($\alpha$=10, err=0.1). We retrain A with $\alpha$=10 on the full dataset, and that's our final model.

Are both approaches correct?
 A: Use nested CV in the following way:

*

*in the inner loop, you optimize both algorithms and their hyperparameters, i.e. A (incl. α) vs. B (incl β)
Basically, what your "without nesting" approach does.
This will give you a readily trained model, i.e. A with some optimal α or B with optimal β


*the outer loop gets you two important internal validation (or rather, verification) results:

*

*is the obtained optimum stable (i.e., is one algorithm consistently preferred, and with reasonably stable hyperparameter)?

*an estimate of generalization performance which both your approaches lack
Keep in mind, though:

*

*it is not necessarily a sensible assumption that one algorithm should outperform another. E.g., if LDA works well, there should be no surprise in finding that logistic regression does so, too.


*Given that your dataset is small, the corresponding uncertainty on your performance estimate may be the limiting factor for the optimization. Make sure you keep an eye on that (and do use proper scoring rules for optimization)
