What's wrong with data-guided modeling in regression? In the Regression Modelling Strategies of Frank Harrell, section 4.1, if I understood correctly, it is not recommendred to using the data to decide how to represent a predictor in a regression model (i.e, to decide predictor complexity and knots' position in a spline), because it could lead to an adequately fitting model, and all statistical measurements of Goodness Of Fit are too good to be true. 
It is said that:

The reason is that the "phantom d.f" that represented assessments are
  forgotten in computing standard errors, P-values, and
  $R^2_{\text{adj}}$



*

*I dont really understand where comes the "phantom d.f" in the computing of above statistics? And how prespecification of predictor complexity can resolve this problem rather than using bootstrap, cross-validation? 

*How can using subject matter knownledge to prespecify predictor complexity can guarantee a model? Sometimes knownledge is obtained from the previous data-driven models' complexity, and one often doesn't have any mean to verify the exactitude of their "knowledge". 

*Is it valid to use data to decide which levels from categorical predictors to be combined? (for ex, levels having more or less the same coefficients and confidence intervals)


Surely there is something wrong with my understanding but I can not figure out what it is.
 A: First, there is no way to "guarantee" any model other than perhaps in physics.  As discussed in detail in my book and Course Notes there is a safe strategy for assigning d.f. to predictors based on a properly masked analysis of "predictive potential" when subject matter knowledge is lacking.  For categorical variables, in the absence of any subject matter knowledge, this may involve combining the least frequent categories so that the total number of categories analyzed equals one more than the d.f. you wish to devote to that predictor.
There is a perfect analogy to help understand the harm done by data-guided modeling.  Suppose you have $k$ groups and wanted to bring evidence for differences in population means between two or more of the groups.  ANOVA with $k-1$ numerator d.f. provides a perfect multiplicity adjustment and yields perfect control of type I error if model assumptions are close to being true.  A data-guided change from this approach would be to decide which of the groups should be pooled before doing a comparison involving fewer than $k-1$ d.f., by combining means that were observed to be "close".  The resulting test would not come close to preserving type I error.  This is exactly the problem with stepwise variable selection or with using the data to tell you how many knots to use for a continuous predictor.  The latter case was studied in detail (using a quadratic fit instead of splines) by Grambsch & O'Brien, whose excellent paper is summarized in my notes.
A: a) hopefully it is clear that standard statistical measures are dealing with the chance of getting a result at  random and do not include the possibilities of you eyeballing the data and using your own particular experience and perception.
b) Prof Harrell doesn't say you couldnt use bootstrap, just that its too much work... you would have to run multiple bootstraps, then for each bootrap data sample  manually eyeball the data decide on your fitting transformation and then run a regression.  Bootstrap is only practical when you have a fully automated process.
