3
$\begingroup$

I do not get the meaning of this sentences from the Elements of Statistical Learning book when talking about subset selection methods before introducing shrinkage methods.

By retaining a subset of the predictors and discarding the rest, subset selection produces a model that is interpretable and has possibly lower prediction error than the full model. However, because it is a discrete process— variables are either retained or discarded— it often exhibits high variance, and so doesn’t reduce the prediction error of the full model.

As far as I understand it is saying that the sub-model possibly has lower prediction error than the full model when considering only the subset of inputs (i.e., when the full model is fed only with the chosen predictors and the others are set to 0?) but why it has higher variance? Should not simpler models have lower variance?

$\endgroup$
4
  • $\begingroup$ For a model type such as "y = f(x)", the simplest model is "y = constant". I personally suspect that this specific model would not usually have a lower variance. If the data is from two sine waves, then a straight line model "y = a + (b * x)" would also not likely have a lower variance. $\endgroup$ Sep 7, 2019 at 11:46
  • 1
    $\begingroup$ In your example: lower variance compared to what? $\endgroup$ Sep 11, 2019 at 8:49
  • $\begingroup$ Lower variance in the residual errors from the regression. $\endgroup$ Sep 11, 2019 at 14:10
  • $\begingroup$ This Q is about a method to use the data itself to select which variables to use in the regression model. That selection will tipically induce its own variance. Search this site for variable selection or model selection, stepwise or subset selection! Many posts ... $\endgroup$ Jun 15, 2021 at 14:07

2 Answers 2

1
$\begingroup$

For the context, here is the full quote:

By retaining a subset of the predictors and discarding the rest, subset selec tion produces a model that is interpretable and has possibly lower predic tion error than the full model. However, because it is a discrete process— variables are either retained or discarded—it often exhibits high variance, and so doesn’t reduce the prediction error of the full model. Shrinkage methods are more continuous, and don’t suffer as much from high variability.

I think subset selection is more aggressive than the lasso, which may discard too many predictors. Thus, the estimated parameters may vary a lot on resampled data. Also, the l1 regularization will also enforce the parameters to be small as well as sparse, but parameters with subset selection may not be small.

$\endgroup$
0
$\begingroup$

I think this variance is with respect to the sampled datasets.

Suppose we have two sampled datasets, i.e., A and B, and 5 predictors. With A, the first predictor is selected. However, with B, the first predictor may not be selected.

Thus, the model as a whole has a high variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.