Grouping predictor factor levels based on response variable I've read that it's bad to do this, but am looking for details as to why.
Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is continuous and $X$s are categorical with many levels. We don't have any opinion on how the levels should be grouped. We then calculate the average response for each level of the predictors, then collapse levels with similar averages.
How badly does this screw up parameter estimates and inferences on them, and why?
 A: This may well depend to an extent on how exactly you will decide which factor levels to pool.
Parameter estimates (e.g., for prediction) may even get better by pooling. Essentially, you are building a more parsimonious model. Your parameter estimates will have lower variance (because of the pooling) but higher bias, this is the ubiquitous bias-variance tradeoff. Your pooling reminds me of trees, which are certainly standard methods for prediction (with extensions, like random forests).
Inferences will probably be a little more tricky. Standard theory is not valid any more, since you are transforming your data after estimation. So don't look for $t$ tables to read off $p$ values. However, you are not filtering on low $p$ values, nor looking for "optimal cutpoints", but explicitly pooling levels with similar parameter estimates - so this could actually not be too far away from the standard tables.
If you do decide to go this way - as I said, this method could actually improve the predictive performance of your model, I recommend that you do some bootstrapping to assess how variable your parameter pooling is.
