Variable selection and validation dataset

Question

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables using validation or cross-validation. For instance, SAS enterprise miner allows us to select variables based on validation data r2 values, and it seems that this is the method SAS encourages us to use. What's the difference here?

Answer 1

A lot of confusion comes from using the term "validation" and in particular "validation set" with different meanings.

Validation e.g. in an engineering context usually just means to measure model/method quality in order to show that the model/method is fit for purpose. This is typically done by measuring the prediction quality of the model.

However, there are also other things you can do with the prediction performance measurement besides reporting whether the model is fit for purpose. For example, you can compare several models wrt. their predictive quality. The crucial point here is that if you use the prediction performance measurement for model selection, this is really part of the training of the final model. Nevertheless, as the procedure of measuing the prediction performance is so far exactly the same regardless what you do later on with those results, it is still called cross validation, even though it is not used for validation.

Having the "validation" as part of the model training means you need to measure that final model's predictive performance on still independent data in order to validate (ìn the engineering meaning) the final model.

See e.g. Feature selection and cross-validation for more details.