Question about training/validation/test

Question

In the book "The Elements of Statistical Learning" at page 222, the authors describe the training/validation/test approach. I find the following passage ambiguous:

Typically our model will have a tuning parameter or parameters $\alpha$ and so we can write our predictions as $\hat f_{\alpha}(x)$. The tuning parameter varies the complexity of our model, and we wish to find the value of $\alpha$ that minimizes error, that is, produces the minimum of the average test error curve in Figure 7.1. Having said this, for brevity we will often suppress the dependence of $\hat f(x)$ on $\alpha$.

It is important to note that there are in fact two separate goals that we might have in mind:

Model selection: estimating the performance of different models in order to choose the best one.

Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

If we are in a data-rich situation, the best approach for both problems is to randomly divide the dataset into three parts: a training set, a validation set, and a test set. The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis. Suppose instead that we use the test-set repeatedly, choosing the model with smallest test-set error. Then the test set error of the final chosen model will underestimate the true test error, sometimes substantially.

It is not clear to me which subset of the data should be used to perform parameter tuning. I think that the authors are omitting that parameter tuning should be performed on the training set. So, I'd say that one way to perform it is to split the training set into two parts (training1 and training2), fit the same model but with different tuning parameters on training1, select the best tuning parameter using training2 and finally fit the model with the optimal tuning parameter on the whole training set. Another way would be to perform crossvalidation on the training set and then fit the model with the optimal tuning parameter on the whole training set. Is my understanding correct?

Answer 1

As you say, parameter tuning is frequently done by selecting the apparently best model from a number of models generated by varying the parameters in question. This is the "model selection" in the cited text.

Your approach is fine as far as it goes, but it does not yet contain the assessment of the final model, which needs independent test cases as well.

And yes, you can do the whole procedure by nesting e.g. cross validation: the inner CV for selecting the best parameters, outer CV to assess the performance for the model trained using parameters selected by the inner CV selection rule.