# Why do subset selection linear models have higher variance than the full model?

I do not get the meaning of this sentences from the Elements of Statistical Learning book when talking about subset selection methods before introducing shrinkage methods.

By retaining a subset of the predictors and discarding the rest, subset selection produces a model that is interpretable and has possibly lower prediction error than the full model. However, because it is a discrete process— variables are either retained or discarded— it often exhibits high variance, and so doesn’t reduce the prediction error of the full model.

As far as I understand it is saying that the sub-model possibly has lower prediction error than the full model when considering only the subset of inputs (i.e., when the full model is fed only with the chosen predictors and the others are set to 0?) but why it has higher variance? Should not simpler models have lower variance?

• For a model type such as "y = f(x)", the simplest model is "y = constant". I personally suspect that this specific model would not usually have a lower variance. If the data is from two sine waves, then a straight line model "y = a + (b * x)" would also not likely have a lower variance. Sep 7, 2019 at 11:46
• In your example: lower variance compared to what? Sep 11, 2019 at 8:49
• Lower variance in the residual errors from the regression. Sep 11, 2019 at 14:10
• This Q is about a method to use the data itself to select which variables to use in the regression model. That selection will tipically induce its own variance. Search this site for variable selection or model selection, stepwise or subset selection! Many posts ... Jun 15, 2021 at 14:07