It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman:
Consider developing a regression model in a context where substantive theory is weak. To focus on an extreme case, suppose that in fact there is no relationship between the dependent variable and the explanatory variables. Even so, if there are many explanatory variables, the R2 will be high. If explanatory variables with small t statistics are dropped and the equation refitted, the R2 will stay high and the overall F will become highly significant. This is demonstrated by simulation and by asymptotic calculation.
Now, say you've got a multiple regression model based on multiple theories; you expect correlation between $y$ and many of your $X$'s, but you're not entirely sure how those correlations are affected by partial co-linearity between your $X$'s. Further, lets say that you suspect that there might be interactions between some of your $X$'s, but you're not sure. Because of collinearity, you want to estimate a model with all the $X$'s, rather than several separate models.
Now, say you were fitting the model to the population, rather that a sample from the population. Suppose that some subset of your $X$'s and your interactions were in fact poor predictors. Since you're in the population, all coefficient estimates are unbiased, and those poor predictions represent real, but weak, correlations.
But what if you're not sure about the model? Coefficient estimates suffer from omitted variables bias, or misspecification bias, or probably both.
Now go from the population to the sample. You specify the same rich model based on your best guess of theory, expecting that some aspects of your theory are correct, and some are not. If you do not select out (say via AIC) some parameters, your model is overfit, and will generalize poorly. If you do select out certain terms (say via backwards selection), your model will suffer from a multiple comparison problem -- certain features will be deemed "significant" even if they do not represent real population correlations.
So what is an applied guy to do? I've got a complicated dataset with a bunch of correlated variables, and a bunch of reasonably well-justified theories. I want to test whether there is any support for the theories, and I want to make predictions (with associated measures of uncertainty of predictions) based on the most robust model I can construct.
I've been doing iterative backwards selection -- dropping terms that lower AIC the most when dropped. Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?
(Note that I do NOT want to do ridge regression or lasso, because I am not willing to trade bias for variance. I know that mis-specification can lead to bias, but I am using semi-parametrics to avoid functional form mis-specification on features.)