I am coming from the field of psychology and in most publications Model Selection (OLS, Regression) is done via Forward/Backward Selection using the F-Static/p-value of the regression coefficients to decide which coefficient to include/exclude in the next step.

Now i read in the statistical learning literature (for example in ESL) that this strategie is outdatet because "it doesn't account for multiple testing issues correctly".

Unluckily there is no further explanation on those problems.

Does anybody can explain me, what exactly is the problem of this approach and why using other criteria (Cp, AIC) is better?

Thanks a lot in advance!

  • 1
    $\begingroup$ I think you will find the information you need in the linked thread. Please read it. If it isn't what you want / you still have a question afterwards, come back here & edit your question to state what you learned & what you still need to know. Then we can provide the information you need without just duplicating material elsewhere that already didn't help you. $\endgroup$ – gung - Reinstate Monica May 21 at 16:00
  • 1
    $\begingroup$ it isn't just overfitting / poor out of sample predictive performance. These methods are highly unlikely to select the right variables. The hypothesis tests are invalid. Etc. To a first approximation, there is nothing good about stepwise selection. $\endgroup$ – gung - Reinstate Monica May 22 at 20:01
  • 1
    $\begingroup$ Regarding the "way out", there isn't really one. This is a very difficult problem. For the most part, you want to think hard, before you conduct a study, about what variables you want to include & why. Ie, variable selection is done before the data exist, & is done based on your knowledge of the topic. If you don't have much relevant knowledge, go get some. Talk to others in the field; read the literature. Etc. You can figure out how much data you can get & how many variables you can afford, & then include a bunch--there's no problem if you have non significant variables in your model. $\endgroup$ – gung - Reinstate Monica May 22 at 20:07
  • 1
    $\begingroup$ After all that, if you really need to do variable selection (ask yourself why you need to do this), you can give up on the idea of hypothesis tests, and use cross-validation and penalization methods. $\endgroup$ – gung - Reinstate Monica May 22 at 20:10
  • 1
    $\begingroup$ @gung-ReinstateMonica thanks for the advice! i was thinking about that now for a couple of days and now i have the feeling that i know more what i am doing. In most cases i am going to build my model based on psychological background knowledge and use ridge regression as some kind of finetuning of the prediction including CV for the hyperparameters. In some cases this is not possible - for example when working with genetic data or a neuro/imaging data - in this cases i will try to refrain from model seletion as long as my n isn't at least as big as my p. $\endgroup$ – platypus May 24 at 22:33