Repeated model selection on bootstrapped data to identify robustness of selected parameters Consider I have a regression model and I want to identify predictor variables that have a significant effect on my dependent variable. (or improve the fit).
I can fit a model with all parameters and do a stepwise backward elimination based on AIC, BIC, Ftest, or else or do LASSO.
No matter what, I obtain a model with reduced parameters, for which I consider the remaining terms as significantly influencing my response .
To check how robust this approach is, I can bootstrap my data and redo the parameter selection.
I do this 1000 times and always note the selected model parameters.
I now have a frequency distribution of how often a term was selected.
To build my final model I now include all terms that are robustly selected in say 99% of the time during bootstrapping.
Is this a valid approach and can I apply corrections for multiple testing here? Say the 99% threshold is arbitrary, but can I interpret the 99% as a p=0.01, collect all p values for each term originally in the model and apply the Benjamini-Hochberg or Bonferroni correction and obtain only terms with say p <0.05?
Update:
The model should be used for inference, identifying parameters that have impact on the response, and not for optimal prediction. It should be parsimonious as possible, so I tend to be conservative in term selection and want to apply the above mentioned p-value correction. Typically the model includes ~7 predictor variables, but first order interactions may be allowed (if this is not making things more complicated) say the full model is:
lm(y~(x1+x2+..x5+fac1+fac2)^2)

Thanks.
 A: Statistical Learning with Sparsity (SLS) covers use of the bootstrap in LASSO and other variable selection in Section 6.2, and continues to discuss inference in Section 6.3. I'd recommend reading that freely available resource closely, and using the related R selectiveInference package for inference, to follow best current practice.
Briefly, it probably makes the most sense to build your model first on the full data set, then illustrate the reliability of the predictor selection via bootstrapping, as in Figure 6.4 of SLS. If you want parsimony, you could use the lambda.1se criterion in selecting the penalty factor, which "gives the most regularized model such that the cross-validated error is within one standard error of the minimum." That keeps fewer predictors than the lambda.min criterion, which is based on the lowest cross-validated error. You document the variability of coefficient estimates among bootstraps and the frequency with which each candidate predictor is omitted from the model.
Inference in this context is a bit tricky. If you have a set of correlated predictors associated with outcome, the particular one(s) selected might change from bootstrap sample to bootstrap sample. In that case no one of them might be selected 99% of the time, but you would still want to have at least one of them maintained in the final model. The approach to inference in SLS and the associated software can document that a particular selected predictor is significantly associated with outcome, even if you can't show that it's "significantly better" than any of its correlated partners.
Finally, given your large number of potential interaction terms, you might want to look at this page for an introduction to the considerations if you maintain interaction terms in a model but omit associated "main effects." This answer includes further references on inference with LASSO.
