tl;dr: If I am using fitting methods that use CV to set their hyper-parameters, is nested CV actually 3 loops deep?
Imagine you have two very simple OLS candidate models: $Y \sim X$ and $Y \sim X + X^2$ and I want to select the one that predicts best.
So there is a simple CV loop I can use, say K-folds:
loop1: split the data into K parts take K-1 parts as training data, last part as testing for each partitioning fit models to training data compute CV error over testing data average CV errors choose model with lowest average CV
Now, that selects my model, but it doesn't tell me what the out of prediction error actually is going to be, so I nest that loop in another CV loop:
loop2: split the data into J parts take J-1 parts as training data, last part as testing for each partitioning perform loop 1 on training compute CV error of model selected by loop1 on testing data average out new CV errors
And that average is a fair estimate of out-of-prediction error
Now same problem, but imagine that rather than having to choose between two OLS models, I have to choose whether to use a (gaussian) Kernel regression or Splines to fit $Y \sim X$. Each of these methods requires CV to set their bandiwth or $\lambda$ penalty.
Do I have to change my loop1 in any way? I understand on the
fit models to training data line I am actually running a further CV model-selection, but is the general loop unchanged? Am I finding the hyper-parameters on 3 times folded data?