Model selection or variable selection first?

Question

What is the right way to pick the best model when you are running multiple models?

I should do variable selection first to find the best subset of variables in each model and then use some model comparison criteria

or

I should use a model comparison criteria first and then do variable selection just in the best one

I'm asking this, because if the second option is valid I would save a little work. However, I do not know if it is possible to guarantee based on the full model that one particular model will be better than the others after selecting variables.

Answer 1

You should not really do either. In order to take the uncertainty around the model and the variables into account, your process will need to do both in parallel. It does not matter so much how you do it, whether it is by bootstrapping your whole process, model averaging, Bayesian approaches with shrinkage priors, some kind of cross-validation approach or any of the other possible approaches. To do anything else gives you overly certain inference (confidence and prediction intervals do not have nominal coverage, p-values are invalid, estimates may be severely biased etc.).