# What's wrong with data-guided modeling in regression?

In the Regression Modelling Strategies of Frank Harrell, section 4.1, if I understood correctly, it is not recommendred to using the data to decide how to represent a predictor in a regression model (i.e, to decide predictor complexity and knots' position in a spline), because it could lead to an adequately fitting model, and all statistical measurements of Goodness Of Fit are too good to be true.

It is said that:

The reason is that the "phantom d.f" that represented assessments are forgotten in computing standard errors, P-values, and $R^2_{\text{adj}}$

1. I dont really understand where comes the "phantom d.f" in the computing of above statistics? And how prespecification of predictor complexity can resolve this problem rather than using bootstrap, cross-validation?
2. How can using subject matter knownledge to prespecify predictor complexity can guarantee a model? Sometimes knownledge is obtained from the previous data-driven models' complexity, and one often doesn't have any mean to verify the exactitude of their "knowledge".
3. Is it valid to use data to decide which levels from categorical predictors to be combined? (for ex, levels having more or less the same coefficients and confidence intervals)

Surely there is something wrong with my understanding but I can not figure out what it is.

There is a perfect analogy to help understand the harm done by data-guided modeling. Suppose you have $k$ groups and wanted to bring evidence for differences in population means between two or more of the groups. ANOVA with $k-1$ numerator d.f. provides a perfect multiplicity adjustment and yields perfect control of type I error if model assumptions are close to being true. A data-guided change from this approach would be to decide which of the groups should be pooled before doing a comparison involving fewer than $k-1$ d.f., by combining means that were observed to be "close". The resulting test would not come close to preserving type I error. This is exactly the problem with stepwise variable selection or with using the data to tell you how many knots to use for a continuous predictor. The latter case was studied in detail (using a quadratic fit instead of splines) by Grambsch & O'Brien, whose excellent paper is summarized in my notes.