From what I understand, variable selection based on p-values (at least in regression context) is highly flawed. It appears variable selection based on AIC (or similar) is also considered flawed by some, for similar reasons, although this seems a bit unclear (e.g. see my question and some links on this topic here: What exactly is "stepwise model selection"?).

But say you do go for one of these two methods to choose the best set of predictors in your model.

Burnham and Anderson 2002 (Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, page 83) state that one should not mix variable selection based on AIC with that based on hypothesis testing: "Tests of null hypotheses and information-theoretic approaches should not be used together; they are very different analysis paradigms."

On the other hand, Zuur et al. 2009 (Mixed Effects Models With Extensions in Ecology with R, page 541) seem to advocate the use of AIC to first find the optimal model, and then perform "fine tuning" using hypothesis testing: "The disadvantage is that the AIC can be conservative, and you may need to apply some fine tuning (using hypothesis testing procures from approach one) once the AIC has selected an optimal model."

You can see how this leaves the reader of both books confused over which approach to follow.

1) Are these just different "camps" of statistical thinking and a topic of disagreement among statisticians? Is one of these approaches simply "outdated" now, but was considered appropriate at the time of writing? Or is one just plain wrong from the start?

2) Would there be a scenario in which this approach would be appropriate? For example, I come from a biological background, where I am often trying to determine which, if any, variables seem to affect or drive my response. I often have a number of candidate explanatory variables and I am trying to find which are "important" (in relative terms). Also, note that the set of candidate predictor variables is already reduced to those considered to have some biological relevance, but this may still include 5-20 candidate predictors.

  • 3
    $\begingroup$ I wonder what Zuur's statistical argument would be for fine tuning with hypothesis testing after AIC selection. It does not seem like a coherent strategy of model building. But I do not know enough about those things. $\endgroup$ – Richard Hardy Mar 15 '17 at 18:19
  • 2
    $\begingroup$ My hunch is that Zuur et al.'s suggestion is bad (why would you ever use significance tests for model selection?), although I'm not sure Burnham and Anderson's statement is correct, either. It's a good question, but I would have to read more deeply of the technical details than I've read so far in order to answer it. $\endgroup$ – Kodiologist Mar 16 '17 at 0:18
  • $\begingroup$ I've used both methods in models to predict panel sales. AIC based stepwise backward regression seemed to give better results from my experience. $\endgroup$ – Souptik Dhar Jun 24 '18 at 15:57
  • 1
    $\begingroup$ @SouptikDhar, when you say "better" results, in which way exactly do you mean? $\endgroup$ – Tilen Jun 27 '18 at 9:48
  • $\begingroup$ Maybe the answer is dependent on the objective of the analysis ? In an observational study, it could be desireable to find the most parcimonious model given the dataset, thus relying on "variable selection based on AIC" for example. However, if the aim is to put an hypothesis to the test, then the model, being a traduction of the hypothesis in term of adequate proxies for the variables of interest to our hypothesis, is already specified from the beginning so there is no room for variable selection IMHO ? $\endgroup$ – Rodolphe Sep 5 '18 at 22:21

A short answer.

The approach of doing data-driven model selection or tuning, then using standard inferential methods on the selected/tuned model (à la Zuur et al., and many other respected ecologists such as Crawley), will always give overoptimistic results: overly narrow confidence intervals (poor coverage), overly small p-values (high type I error). This is because standard inferential methods assume the model is specified a priori; they don't take the model tuning process into account.

This is why researchers like Frank Harrell (Regression Modeling Strategies) strongly disapprove of data-driven selection techniques like stepwise regression, and caution that one must do any reduction of the model complexity ("dimension reduction", e.g. computing a PCA of the predictor variables and selecting the first few PCA axes as predictors) by looking only at the predictor variables.

If you are interested only in finding the best predictive model (and aren't interested in any kind of reliable estimate of the uncertainty of your prediction, which falls in the realm of inference!), then data-driven model tuning is fine (although stepwise selection is rarely the best available option); machine learning/statistical learning algorithms do a lot of tuning to try to get the best predictive model. The "test" or "out-of-sample" error must be assessed on a separate, held-out sample, or any tuning methods need to be built into a cross-validation procedure.

There does seem to have been historical evolution in opinions on this topic; many classic statistical textbooks, especially those that focus on regression, present stepwise approaches followed by standard inferential procedures without taking the effects of model selection into account [citation needed ...]

There are many ways to quantify variable importance, and not all fall into the post-variable-selection trap.

  • Burnham and Anderson recommend summing AIC weights; there's quite a bit of disagreement over this approach.
  • You could fit the full model (with appropriately scaled/unitless predictors) and rank the predictors by estimated magnitude [biological effect size] or Z-score ["clarity"/statistical effect size].

I come from a biological background and am a hired biostatistician, working in in a university hospital. I read alot on this, especially recently, including especially Harrell's opinions over the www, and his book Regression Modelling Strategies. Not quoting him anymore, but speaking from experience: It's highly field related, I think this is the first level that has to be accounted for. Second level would be to get a good rational approach, meaning that your predictors should be core meaningful to express what you want to predict, by scientific experience. 3rd would be to account for interactions, which is super crucial, and can be addressed by the statistical approach taken or insight. Only 4th is the method chosen, in my case with hospital data, that quite often has about x*10^3 datapoints and x*10^1 observations in e.g. logistic or cox regression, I for now tested backwards stepwise AIC, Lasso, Ridge and Elastic Net regression against each other by predictive AUC and have to say: The methods are varying, but stepwise AIC is handling the issue aswell as Lasso and Elastic Net, quite often even better.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.