Internal validation steps

Question

I am currently developing models for a prediction analysis. I have read through much of Harrell's Regression Modeling Strategies text and am confused on one point regarding internal validation.

Specifically, it is stated that you must repeat the modeling steps you used to develop the model in your original sample in the validation sample(s), including tests of linearity, additivity, and variable selection (see ch. 4, pg. 97, step 16). Does this mean that when you re-fit the models in your validation samples that you could potentially end up with different predictors in the model? This idea seems strange to me though my suspicion is that it is because what we are validating is our process of model building rather than the specific model we developed in the original sample. I just want to make sure I am clear on this point before I begin the analysis.

Answer 1

Expanding on the comment from gung - Reinstate Monica:

You have correctly grasped the major point. With this bootstrap approach you are validating your model-building process.

Under the bootstrap principle, resampling from your data set is akin to taking data sets from the underlying population. Thus, if repeating your modeling process on multiple bootstrap samples from the data fits the full data set well, it's reasonable to assume that your modeling process applied to the full data set will reasonably represent the situation in the underlying population.

The instability of variable selection you note is particularly an issue in LASSO. Hastie et al illustrate bootstrap evaluation of LASSO modeling in Section 6.2 of Statistical Learning with Sparsity, with cross-validated choice of penalty and consequent variable selection performed on each resample. They show graphical displays of the distributions of regression-coefficient estimates among the models and of the frequency of omission of each predictor. If your modeling process involves predictor selection, you might want to generate similar displays.

This is one reason why Steps 2 and 6 in Harrell's list are so helpful, quoted here in part:

2. Formulate good hypotheses that lead to specification of relevant candidate predictors and possible interactions.

6. If the number of terms fitted or tested in the modeling process ... is too large in comparison with the number of outcomes in the sample, use data reduction (ignoring Y) until the number of remaining free variables needing regression coefficients is tolerable. ... Alternatively, use penalized estimation with the entire set of variables.

If you start with a well-specified list of predictors based on your understanding of the subject matter, with either a number of predictors appropriate to the size of your data set or penalization like ridge regression that doesn't perform variable selection, there will be no data-driven variable selection to examine in the bootstrapping. That simplifies the validation and calibration via bootstrapping.

Answer 2

Normally, validation sets or validation is done inside of the cross validation function for hyperparameter testing. So, you actually don't care about a specific validation set. You normally end up with train, test.

What I believe is or could be the meaning of the part you quoted, without having access to the text of yours, is, that when we model the outcome of different estimators, may it be different regressions and when applying different strategies to deal with the data for regression, that we make sure all techniques our data undergoes is applied to every subsample of the cv in the same way AND that the subsamples for all estimators are even so that we can really compare performance. This is the reason why we normally not do this by manually. So in other words what you do, make sure you do it for 'everyone' in the same way. No 'one' should be treated unequally.

Does this help you or do I misinterpret your question?