1
$\begingroup$

What is the right way to pick the best model when you are running multiple models?

I should do variable selection first to find the best subset of variables in each model and then use some model comparison criteria

or

I should use a model comparison criteria first and then do variable selection just in the best one

I'm asking this, because if the second option is valid I would save a little work. However, I do not know if it is possible to guarantee based on the full model that one particular model will be better than the others after selecting variables.

$\endgroup$
3
$\begingroup$

You should not really do either. In order to take the uncertainty around the model and the variables into account, your process will need to do both in parallel. It does not matter so much how you do it, whether it is by bootstrapping your whole process, model averaging, Bayesian approaches with shrinkage priors, some kind of cross-validation approach or any of the other possible approaches. To do anything else gives you overly certain inference (confidence and prediction intervals do not have nominal coverage, p-values are invalid, estimates may be severely biased etc.).

$\endgroup$
  • 1
    $\begingroup$ Thanks for you reply, this question is related with this one that you already answered too stats.stackexchange.com/questions/280057/…. I applied a Bayesian lasso with double exponential priori in mean and dispersion parameters, but I'm doubt with this type of approach since I get bigger DIC, BIC, Deviance values in a model with lasso than with Gaussian priors. $\endgroup$ – user72621 May 22 '17 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy