Is inference based on full (global) regression model appropriate?

Question

Is inference based on a full model appropriate, and if so, in which circumstances?

Suppose you are interested in the potential relationship between a response variable and several candidate predictor variables, and use some form of regression (e.g. generalized linear model) to answer that. One approach to inferring which predictors are "important" or have an apparently genuine relationship with the response would be information-theoretic criterion (say AIC) based model comparison. Even though variables that are not retained in the final model might have some relationship with the response, they essentially provide no additional substantial information, given other predictors retained in the model.

Is there a case where it would be more appropriate to simply fit a full (global) model (with all candidate predictors), stop there, and base inferences on individual predictors solely on the t-statistics (or other statistics) and p-values in this full model, without further model selection?

I have come across suggestions (e.g. Whittingham et al. "Why do we still use stepwise modelling in ecology and behaviour?" (2006) that this might be a sensible thing to do, albeit with potential drawbacks. The authors say that estimated parameters are unbiased, but other sources say that these estimates and p-values are not to be trusted, as other ("non-important") variables in the model may affect them.

If the aim is to understand potential biological relationships, which method would be more appropriate?

Answer 1

All depends on your study aims:

A) Exploratory study: Your aim is to screen a number of potentially interesting predictors for relationships. You want to to build a testable model based on these exploratory results. No inferences (in a null-hypothesis-significance-testing sense), or other important decisions are drawn from the study. The study is a pilot and will be followed by another confirmatory/prespecified study. In this case model selection procedures (using AIC, BIC, or cross-validation techniques) are your methods of choice. The reference you cited is correct: The p-Values obtained for the predictors in the final model will be overly optimistic: By essentially trying out many different models in model selection you created a multiple-comparisons-problem — "the garden of forking paths". Conventional statistical tests will yield you p-values for the current model only and not control for these multiple comparisons.

B) Confirmatory/"pre-specified" study: In this case you should ideally test a single model — the one pre-specified before the study was performed. If you had good reason to believe before the study started that all of your predictors are having an effect then the full model is a natural choice. If you included some predictors on mere suspicion you likely performed an exploratory study.

"Non-important" variables, i.e. variables that do not explain much variance in the outcome variable, will only exert undue influence on your data if you have too many predictors relative to your sample size (overfitting) or if there are predictors that are highly correlated (collinear). Ideally you avoid these situations by performing an exploratory study.

One way to check for overfitting/unstable model problems is by exploring a "reduced model" that includes "significant" terms from the main model, only. Importantly, this reduced model analysis should be referred to as a post-hoc control analysis aiding interpretation. Conclusions should solely be based on the pre-specified model.