In the Regression Modelling Strategies of Frank Harrell, section 4.1, if I understood correctly, it is not recommendred to using the data to decide how to represent a predictor in a regression model (i.e, to decide predictor complexity and knots' position in a spline), because it could lead to an adequately fitting model, and all statistical measurements of Goodness Of Fit are too good to be true.

It is said that:

The reason is that the "phantom d.f" that represented assessments are forgotten in computing standard errors, P-values, and $R^2_{\text{adj}}$

  1. I dont really understand where comes the "phantom d.f" in the computing of above statistics? And how prespecification of predictor complexity can resolve this problem rather than using bootstrap, cross-validation?
  2. How can using subject matter knownledge to prespecify predictor complexity can guarantee a model? Sometimes knownledge is obtained from the previous data-driven models' complexity, and one often doesn't have any mean to verify the exactitude of their "knowledge".
  3. Is it valid to use data to decide which levels from categorical predictors to be combined? (for ex, levels having more or less the same coefficients and confidence intervals)

Surely there is something wrong with my understanding but I can not figure out what it is.


2 Answers 2


First, there is no way to "guarantee" any model other than perhaps in physics. As discussed in detail in my book and Course Notes there is a safe strategy for assigning d.f. to predictors based on a properly masked analysis of "predictive potential" when subject matter knowledge is lacking. For categorical variables, in the absence of any subject matter knowledge, this may involve combining the least frequent categories so that the total number of categories analyzed equals one more than the d.f. you wish to devote to that predictor.

There is a perfect analogy to help understand the harm done by data-guided modeling. Suppose you have $k$ groups and wanted to bring evidence for differences in population means between two or more of the groups. ANOVA with $k-1$ numerator d.f. provides a perfect multiplicity adjustment and yields perfect control of type I error if model assumptions are close to being true. A data-guided change from this approach would be to decide which of the groups should be pooled before doing a comparison involving fewer than $k-1$ d.f., by combining means that were observed to be "close". The resulting test would not come close to preserving type I error. This is exactly the problem with stepwise variable selection or with using the data to tell you how many knots to use for a continuous predictor. The latter case was studied in detail (using a quadratic fit instead of splines) by Grambsch & O'Brien, whose excellent paper is summarized in my notes.


a) hopefully it is clear that standard statistical measures are dealing with the chance of getting a result at random and do not include the possibilities of you eyeballing the data and using your own particular experience and perception.

b) Prof Harrell doesn't say you couldnt use bootstrap, just that its too much work... you would have to run multiple bootstraps, then for each bootrap data sample manually eyeball the data decide on your fitting transformation and then run a regression. Bootstrap is only practical when you have a fully automated process.

  • $\begingroup$ that is to say, pooling levels by eyeballing the data is still meaningful (taking into account confidence intervals before pooling), except that the posteriori statistical measures are not valid anymore? $\endgroup$
    – Metariat
    Mar 30, 2016 at 8:37
  • $\begingroup$ @frankharrell I think Metallica wanted to comment on your answer $\endgroup$
    – seanv507
    Apr 5, 2016 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.