# Reference request: accounting for AIC stepwise model selection with bootstrapped standard errors

The use of the AIC for model selection via comparison over many models is well-known and rightly maligned. But I read an interesting passage from Hastie et. al.'s classic The Elements of Statistical Learning:

... It is ... tempting after a model search to print out a summary of the chosen model... however, the standard errors are not valid, since they do not account for the search process. The bootstrap... can be useful in such settings.

I'm intrigued by this line. The authors then later describe how this procedure would be done, but in the context of $$B$$-splines: incorporate the selection process into the bootstrap procedure. Anyone know of references expanding on the properties of this procedure? The book doesn't mention any.

Chapter 6 of Statistical Learning with Sparsity by Hastie et al covers inference in models that are based on predictor selection in some detail, with references to the literature.

Section 6.2 uses the bootstrap in the context of LASSO, but the approach is quite general. Under the bootstrap principle--resampling from your data set represents sampling from the underlying population--repeating every step of the modeling on multiple bootstrap samples provides estimates of the distributions of regression-coefficient estimates. Testing the models against the full data sample then validates your modeling approach in terms of how well it should work on the underlying population. This process also provides an estimate of bias, which can be used to correct the original model results. That's a useful way to validate any modeling approach.

As Hastie et al illustrate for LASSO, repeating "every step of the modeling" means selection of the penalty factor to minimize cross-validated error, and accepting the corresponding retained predictors and penalized coefficients, on each bootstrap sample. The predictors retained by LASSO models will tend to differ among bootstrap samples, so one can use stability plots to illustrate how frequently each predictor is retained.

If you wish to use stepwise selection instead of LASSO, repeating "every step of the modeling" would be building a fresh stepwise model on each bootstrap sample. As with LASSO, the particular predictors retained would differ among models; you could produce stability plots and estimated distributions of coefficient values as for LASSO. Unlike LASSO, the predictor coefficients would not be penalized. You might find it interesting to compare LASSO with standard stepwise selection on the same data sets, evaluating both approaches with the bootstrap validation outlined above.

You should be aware of other ways to correct for predictor selection. Hastie et al, in Section 6.3.2, use a different approach that does not rely on bootstrapping, instead developing a "spacing test" that provides:

a general scheme for inference after selection—-one that yields exact p-values and confidence intervals in the Gaussian case. It can deal with any procedure for which the selection events can be characterized by a set of linear inequalities in [outcome] y.

As discussed in Section 6.3.3, at each step after the first this latter approach "tests whether the partial correlation of the given predictor entered at that step is zero, adjusting for other variables that are currently in the model," so care is needed in interpretation. With that caveat, "the general inference procedure outlined in Section 6.3.2 can in fact be applied to forward stepwise regression, providing proper selective inference for that procedure as well."

Non-bootstrap methods for inference in variable-selection models are implemented in the R selectiveInference package, part of a broader selective inference software project.