Expanding on the comment from gung - Reinstate Monica:
You have correctly grasped the major point. With this bootstrap approach you are validating your model-building process.
Under the bootstrap principle, resampling from your data set is akin to taking data sets from the underlying population. Thus, if repeating your modeling process on multiple bootstrap samples from the data fits the full data set well, it's reasonable to assume that your modeling process applied to the full data set will reasonably represent the situation in the underlying population.
The instability of variable selection you note is particularly an issue in LASSO. Hastie et al illustrate bootstrap evaluation of LASSO modeling in Section 6.2 of Statistical Learning with Sparsity, with cross-validated choice of penalty and consequent variable selection performed on each resample. They show graphical displays of the distributions of regression-coefficient estimates among the models and of the frequency of omission of each predictor. If your modeling process involves predictor selection, you might want to generate similar displays.
This is one reason why Steps 2 and 6 in Harrell's list are so helpful, quoted here in part:
- Formulate good hypotheses that lead to specification of relevant candidate predictors and possible interactions.
- If the number of terms fitted or tested in the modeling process ... is too large in comparison with the number of outcomes in the sample, use data reduction (ignoring Y) until
the number of remaining free variables needing regression coefficients is
tolerable. ... Alternatively, use penalized estimation
with the entire set of variables.
If you start with a well-specified list of predictors based on your understanding of the subject matter, with either a number of predictors appropriate to the size of your data set or penalization like ridge regression that doesn't perform variable selection, there will be no data-driven variable selection to examine in the bootstrapping. That simplifies the validation and calibration via bootstrapping.