Suppose that we have 3 different regression/classification methods: $f_1(D,\alpha)$, $f_2(D,\alpha)$, $f_3(D,\alpha)$ (for instance: lasso, neural network and SVM) where $D$ is the dataset and $\alpha$ is the (not yet chosen) tuning parameter (or the vector of tuning parameters) of the method. The goals are:
- to identify the best class of models for the specific dataset
- to identify the optimal values of the tuning parameters of the method selected at the previous step
I use the word "class" in the sense that $f_1(D,\alpha_1)$,$f_1(D,\alpha_2)$,$f_1(D,\alpha_3)$ are all members of the same class of models $f_1(D,-)$.
So, we first apply nested cross validation on the whole dataset to $f_1(D,\alpha)$, then to $f_2(D,\alpha)$, and finally to $f_3(D,\alpha)$ (the inner cross validation obviously tries different values of $\alpha$ for each class of models). The outer cross validation gives us an estimate of the prediction error for each class of models and we select the class of model which produces the minimum error. Suppose that $f_2(D,-)$ is the winning class of models. The nested cross validation gives us $k$ models (where $k$ is the number of folds of the outer cross validation): $f_2(D,\alpha_1)$,$f_2(D,\alpha_2)$,...,$f_2(D,\alpha_k)$ where $\alpha_1$, $\alpha_2$,...$\alpha_k$ can be different. Now that we know that $f_2(D,-)$ is the best class of models, we need to select the optimal values for its tuning parameters.
The questions are:
- is it correct to perform parameter tuning on the whole dataset after nested cross validation to select the optimal tuning parameters of the winning class of models?
- is the estimate of prediction error produced by the outer cross validation still unbiased after that or does it get optimistic?
- if performing parameter tuning on the whole dataset after nested cross validation is not correct, how do I select a final model for prediction/inference?