It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman:

Consider developing a regression model in a context where substantive theory is weak. To focus on an extreme case, suppose that in fact there is no relationship between the dependent variable and the explanatory variables. Even so, if there are many explanatory variables, the R2 will be high. If explanatory variables with small t statistics are dropped and the equation refitted, the R2 will stay high and the overall F will become highly significant. This is demonstrated by simulation and by asymptotic calculation.

Now, say you've got a multiple regression model based on multiple theories; you expect correlation between $y$ and many of your $X$'s, but you're not entirely sure how those correlations are affected by partial co-linearity between your $X$'s. Further, lets say that you suspect that there might be interactions between some of your $X$'s, but you're not sure. Because of collinearity, you want to estimate a model with all the $X$'s, rather than several separate models.

Now, say you were fitting the model to the population, rather that a sample from the population. Suppose that some subset of your $X$'s and your interactions were in fact poor predictors. Since you're in the population, all coefficient estimates are unbiased, and those poor predictions represent real, but weak, correlations.

But what if you're not sure about the model? Coefficient estimates suffer from omitted variables bias, or misspecification bias, or probably both.

Now go from the population to the sample. You specify the same rich model based on your best guess of theory, expecting that some aspects of your theory are correct, and some are not. If you do not select out (say via AIC) some parameters, your model is overfit, and will generalize poorly. If you do select out certain terms (say via backwards selection), your model will suffer from a multiple comparison problem -- certain features will be deemed "significant" even if they do not represent real population correlations.

So what is an applied guy to do? I've got a complicated dataset with a bunch of correlated variables, and a bunch of reasonably well-justified theories. I want to test whether there is any support for the theories, and I want to make predictions (with associated measures of uncertainty of predictions) based on the most robust model I can construct.

I've been doing iterative backwards selection -- dropping terms that lower AIC the most when dropped. Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?

(Note that I do NOT want to do ridge regression or lasso, because I am not willing to trade bias for variance. I know that mis-specification can lead to bias, but I am using semi-parametrics to avoid functional form mis-specification on features.)

  • $\begingroup$ If you have the data to support it, you can do things like sample splitting, where a model identified on one subset of the data, estimated on a second subset and evaluated for predictive ability on a third subset. Carefully applied, cross-validation can be used as another way to approach the need for working 'out-of-sample' - to not do everything on the same data. Alternatively, you need to evaluate the properties of your whole procedure from start to finish when considering bias in estimates or predicted values, their standard errors, type I error rates, power, etc etc (e.g. by simulation) $\endgroup$ – Glen_b May 1 '13 at 7:08
  • $\begingroup$ your bias-variance comment does not make sense in light of model selection. By subsetting the predictors, then you are trading variance for bias. $\endgroup$ – probabilityislogic May 1 '13 at 8:52
  • $\begingroup$ @Glen_b: cross validation might be a bit too-data intensive for my multi-level/clustered data. Not sure I understand the second half of your comment. $\endgroup$ – generic_user May 1 '13 at 12:19
  • $\begingroup$ @problogic: I suppose you've got a point. But will the bias engendered by removing a variable be similar to the bias engendered by applying a shrinkage penalty to each term? Wouldn't the ridge bias be larger? I've never really used regularization estimators much, so I'm a bit leery of them. $\endgroup$ – generic_user May 1 '13 at 12:22
  • $\begingroup$ The second half was saying that if you ignore the problem with using the same data for the different parts (model identification, parameter estimation, and model evaluation), and just use the same data for all three, you should understand the properties of the things you do calculate - and one way to do that is by simulation. $\endgroup$ – Glen_b May 1 '13 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.