When is it justified to "peek" at the outcome variable in model-building process?

Question

I am referring to the following comment made in a 1996 paper by Dr Frank Harrell et al in Statistics in Medicine:

Unless a formal penalized estimation technique is used, multiple comparisons problems that arise from ‘peeking’ at the outcome variable must be eliminated; data reduction methods must be used that do not utilize the outcome variable.

I took this to mean the following two:

1. One should not use graphical or informal analyses to guide the analysis. Hence it is not advisable to look at graphical plot of outcome variable vs covariates and use the result to inform the choice of model.
2. One should not drop variables from model just because p values for those variables are not significant.

However, what I see in the Case Study in the lecture note (http://hbiostat.org/rmsc) of Dr Harrell seems to suggest that my understanding is incorrect. In particular,

1. He performed a couple non-parametric regression estimates of some variables with the outcome variable (survive or not). And then concluded that "Insufficient variation in sibsp, parch to fit complex interactions or nonlinearities." (p.260) This seems to be in direct contradiction of my understanding #1, or maybe this is drawn from some other information, not the plots themselves? What information may we safely extract from graphical or other informal analyses, without the risk of introducing bias?
2. On p.262, after fitting a saturated model, he made the comment that "parch clearly insignificant, so drop". This seems to go against my understanding #2. What am I missing in my reading? When is it ok to use regression results (or any other data reduction method utilizing the outcome variable) to make decision about dropping variables / simplification?

Thank you very much!

Answer 1

I think part of the problem here is that the lecture notes are too cryptic. The recently released second edition of Dr. Harrell's Regression Modeling Strategies provides important missing detail on this analysis of survival from the Titanic, on pages 292-300.

For question 1, there is no problem in looking at plots, distributions of variables, etc., to select predictor variables provided that the outcome variable is not consulted. That's actually an important part of the model building process if there are too many predictors for the number of cases, as covered extensively in Chapter 4 of the book. The following quote from page 292 of the book clarifies the issue in this particular model:

The sibsp and parch variables do not have sufficiently dispersed distributions to allow for us to model them nonlinearly. Also, there are too few passengers with nonzero values of these two variables in sex × pclass × age strata to allow us to model complex interactions involving them. The meaning of these variables does depend on the passenger’s age, so we consider only age interactions involving sibsp and parch

So no information was used here about the outcome variable.

For question 2, the continuation on page 292 says:

Three-way interactions are clearly insignificant (P = 0.4) in Table 12.1. So is parch (P = 0.6 for testing the combined main effect + interaction effects for parch, i.e., whether parch is important for any age). These effects would be deleted in almost all bootstrap resamples had we bootstrapped a variable selection procedure using α = 0.1 for retention of terms, so we can safely ignore these terms for future steps.

This is where professional judgment and the intended purpose of the modeling comes into play. Nothing requires that all possible predictor variables be included in a model. Best performance of predictive models supports inclusion of variables that do not reach "statistical significance." But as noted on page 299:

There will never be another Titanic, so we do not need to validate the model for prospective use.

For some applications there may be an advantage to cutting down somewhat on the number of predictors. Note the following last step in building (as opposed to validating) a "final" model, from page 97 of the book:

13. Do limited backwards step-down variable selection if parsimony is more important than accuracy. The cost of doing any aggressive variable selection is that the variable selection algorithm must also be included in a resampling procedure to properly validate the model or to compute confidence limits and the like.

Pages 299-300 include a relevant note on validating, given that parch and 3-way interactions were previously removed:

But we use the bootstrap to validate the model anyway, in an effort to detect whether it is overfitting the data. We do not penalize the calculations that follow for having examined the effect of parch or or testing three-way interactions, in the belief that these tests would replicate well.

I'd recommend getting a copy of the full book if this type of model building is important to you.