I am going through the lab portion on variable subset selection methods in An Introduction to Statistical Learning. On page 249 it says:

Finally, we perform best subset selection on the full data set, and select the best ten-variable model. It is important that we make use of the full data set in order to obtain more accurate coefficient estimates. Note that we perform best subset selection on the full data set and select the best ten- variable model, rather than simply using the variables that were obtained from the training set, because the best ten-variable model on the full data set may differ from the corresponding model on the training set.

I am a bit confused on this point because it would seem that we should use the validation approach to select the particular variables that give us the lowest test error, then estimate the final model using those variables on the full data set. Here, it seems that we are only using the validation approach to select the number of variables that should be in the final model, and then using the full data set to select the particular variables.

What is the advantage of selecting the particular variables using the full data set? I understand why we would want to estimate the coefficients for the final model using as many data points as possible. But isn't the particular variables to include (and not just the number of variables) the kind of hyper-parameters that we would want to tune using a validation approach?


I take it from your question that you understand the need for splitting the data into training and test sets, but are confused about why the best subset selection model is implemented in the way it is...

The best subset selection model simply aims to select the best $n$ (my notation, not theirs) covariates.

Instead, you could define a model that includes 3 boolean hyperparameters for your baseball example:

  • include variable "AtBat"?
  • include variable "Hits"?
  • include variable "Walks"?

giving you a total of $2^3=8$ different combinations of variables. In fact, such a model wouldn't be that hard to code up on your own, as to tune it you'd simply have to loop through the 8 different combinations (for a naive implementation) on the training/test set split. And it would be applicable to other datasets that had variables of that name.

But it would not be applicable to other cases where variables are named differently. And even on this specific example, it's easy to see that it's a little unwieldy.

So it makes sense to limit the model to a single hyperparameter, which is the number $n$ of variables to select.

Therefore, because your model has that single hyperparameter, that is the only thing you can vary during the tuning process.

Like you say, it's very possible (for a fixed value of the hyperparameter) that the model may select different variables under the hood when it is trained on different datasets, or indeed different subsets or splits of the same dataset. The greater the "variance" of the model (in terms of the bias-variance tradeoff), the more likely this is to happen. Something similar happens with tree-based models, for example.

Still, that's nothing to worry about. If it's a good model, that shouldn't matter - the additional data may help it achieve a better performance out-of-sample anyway.

By the way, if you're interested in these topics I heartily recommend Chapter 7 of ESL.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.