1
$\begingroup$

In the book "The Elements of Statistical Learning" at page 222, the authors describe the training/validation/test approach. I find the following passage ambiguous:

Typically our model will have a tuning parameter or parameters $\alpha$ and so we can write our predictions as $\hat f_{\alpha}(x)$. The tuning parameter varies the complexity of our model, and we wish to find the value of $\alpha$ that minimizes error, that is, produces the minimum of the average test error curve in Figure 7.1. Having said this, for brevity we will often suppress the dependence of $\hat f(x)$ on $\alpha$.

It is important to note that there are in fact two separate goals that we might have in mind:

Model selection: estimating the performance of different models in order to choose the best one.

Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

If we are in a data-rich situation, the best approach for both problems is to randomly divide the dataset into three parts: a training set, a validation set, and a test set. The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis. Suppose instead that we use the test-set repeatedly, choosing the model with smallest test-set error. Then the test set error of the final chosen model will underestimate the true test error, sometimes substantially.

It is not clear to me which subset of the data should be used to perform parameter tuning. I think that the authors are omitting that parameter tuning should be performed on the training set. So, I'd say that one way to perform it is to split the training set into two parts (training1 and training2), fit the same model but with different tuning parameters on training1, select the best tuning parameter using training2 and finally fit the model with the optimal tuning parameter on the whole training set. Another way would be to perform crossvalidation on the training set and then fit the model with the optimal tuning parameter on the whole training set. Is my understanding correct?

$\endgroup$
2
$\begingroup$

As you say, parameter tuning is frequently done by selecting the apparently best model from a number of models generated by varying the parameters in question. This is the "model selection" in the cited text.

Your approach is fine as far as it goes, but it does not yet contain the assessment of the final model, which needs independent test cases as well.

And yes, you can do the whole procedure by nesting e.g. cross validation: the inner CV for selecting the best parameters, outer CV to assess the performance for the model trained using parameters selected by the inner CV selection rule.

$\endgroup$
  • $\begingroup$ Thank you. So the validation set is used for parameter tuning while the test set is used to assess the performance of the model, with the optimal tuning parameter, fitted on the set of data given by the union between the training and the validation set. Is it correct? Suppose that I have to select the best model between a neural network and a support vector machine for a given prediction problem. Both of them have a vector of tuning parameters. I use the validation set to find the best vector of tuning parameters for the neural network and for the SVM. (continues) $\endgroup$ – John M Mar 10 '16 at 16:04
  • $\begingroup$ Then I fit the neural network on the training+validation set with the optimal vector of tuning parameters and I do the same for the SVM. This leads to two final models (a NN and a SVM) whose performance is assessed with the test set. Finally I choose the final model between the two with the minimum prediction error in the test set. So is it correct to say that the test set is used to choose the winning final model? $\endgroup$ – John M Mar 10 '16 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.