Variance: best-subset vs forward-stepwise regression

Question

In Hastie's Elements of Statistical Learning, in the chapter about linear regression, it is stated that

A price is paid in variance for selecting the best subset of each size; forward-stepwise is a more constrained search, and will have lower variance but perhaps more bias.

This seems intuitive, as best-subset can result in overfitting. Nevertheless, I don't know how to prove mathematically that the model obtained with forward-stepwise has lower variance than the one given by best-subset. Can this be proven, or are the authors just providing some heuristics ?

Answer 1

When it comes to the residual variance, a model selected via best subset will trivially better, because the best model is the one which minimizes the residual standard errors for a specific model size.

Thus it is pretty clear that the authors are referring to the variability in which model is selected. This variance is not well defined in the paragraph. Intuitively, but subset selection considers a larger set of models and therefore is more variable to the step-wise procedure which only considers a single model of every size. On the other hand, step-wise selection might not have the correct model in its solution path which is where the bias would come from.