Is it correct to perform parameter tuning after nested cross validation?

Question

Suppose that we have 3 different regression/classification methods: $f_1(D,\alpha)$, $f_2(D,\alpha)$, $f_3(D,\alpha)$ (for instance: lasso, neural network and SVM) where $D$ is the dataset and $\alpha$ is the (not yet chosen) tuning parameter (or the vector of tuning parameters) of the method. The goals are:

• to identify the best class of models for the specific dataset
• to identify the optimal values of the tuning parameters of the method selected at the previous step

I use the word "class" in the sense that $f_1(D,\alpha_1)$,$f_1(D,\alpha_2)$,$f_1(D,\alpha_3)$ are all members of the same class of models $f_1(D,-)$.

So, we first apply nested cross validation on the whole dataset to $f_1(D,\alpha)$, then to $f_2(D,\alpha)$, and finally to $f_3(D,\alpha)$ (the inner cross validation obviously tries different values of $\alpha$ for each class of models). The outer cross validation gives us an estimate of the prediction error for each class of models and we select the class of model which produces the minimum error. Suppose that $f_2(D,-)$ is the winning class of models. The nested cross validation gives us $k$ models (where $k$ is the number of folds of the outer cross validation): $f_2(D,\alpha_1)$,$f_2(D,\alpha_2)$,...,$f_2(D,\alpha_k)$ where $\alpha_1$, $\alpha_2$,...$\alpha_k$ can be different. Now that we know that $f_2(D,-)$ is the best class of models, we need to select the optimal values for its tuning parameters.

The questions are:

1. is it correct to perform parameter tuning on the whole dataset after nested cross validation to select the optimal tuning parameters of the winning class of models?
2. is the estimate of prediction error produced by the outer cross validation still unbiased after that or does it get optimistic?
3. if performing parameter tuning on the whole dataset after nested cross validation is not correct, how do I select a final model for prediction/inference?

Thank you.

Answer 1

Yes, you should perform hyper-parameter tuning on the whole dataset for the best model that was selected by nested cross validation. The reason is you want to make your final model as good as possible (since generally, the more the training data, the better the outcome model). So the following procedure is (after nested cross validation),

1. Do cross validation on the whole dataset using your best model. Choose one best hyperparameter.

2. Train your best model again on the whole dataset using the best hyper-parameter. The model you get is the final model.

If the future data distributes the same as your data currently in hand, in average (you do such things many times), your final model should perform better than that shown by the outer cross validation in model selection. This is because it was trained on more data.