Is inference based on a full model appropriate, and if so, in which circumstances?

Suppose you are interested in the potential relationship between a response variable and several candidate predictor variables, and use some form of regression (e.g. generalized linear model) to answer that. One approach to inferring which predictors are "important" or have an apparently genuine relationship with the response would be information-theoretic criterion (say AIC) based model comparison. Even though variables that are not retained in the final model might have some relationship with the response, they essentially provide no additional substantial information, given other predictors retained in the model.

Is there a case where it would be more appropriate to simply fit a full (global) model (with all candidate predictors), stop there, and base inferences on individual predictors solely on the t-statistics (or other statistics) and p-values in this full model, without further model selection?

I have come across suggestions (e.g. Whittingham et al. "Why do we still use stepwise modelling in ecology and behaviour?" (2006) that this might be a sensible thing to do, albeit with potential drawbacks. The authors say that estimated parameters are unbiased, but other sources say that these estimates and p-values are not to be trusted, as other ("non-important") variables in the model may affect them.

If the aim is to understand potential biological relationships, which method would be more appropriate?

  • $\begingroup$ Please provide the "other sources" mentioned to help improving the answer below. $\endgroup$ – mzunhammer Mar 14 '17 at 19:13
  • $\begingroup$ @mzunhammer, I wanted to do that in the first place, but was unable to track down those sources again. I can't remember in which paper I read that, and could not find it yet. :) Will keep searching though. $\endgroup$ – Tilen Mar 15 '17 at 17:10

All depends on your study aims:

A) Exploratory study: Your aim is to screen a number of potentially interesting predictors for relationships. You want to to build a testable model based on these exploratory results. No inferences (in a null-hypothesis-significance-testing sense), or other important decisions are drawn from the study. The study is a pilot and will be followed by another confirmatory/prespecified study. In this case model selection procedures (using AIC, BIC, or cross-validation techniques) are your methods of choice. The reference you cited is correct: The p-Values obtained for the predictors in the final model will be overly optimistic: By essentially trying out many different models in model selection you created a multiple-comparisons-problem — "the garden of forking paths". Conventional statistical tests will yield you p-values for the current model only and not control for these multiple comparisons.

B) Confirmatory/"pre-specified" study: In this case you should ideally test a single model — the one pre-specified before the study was performed. If you had good reason to believe before the study started that all of your predictors are having an effect then the full model is a natural choice. If you included some predictors on mere suspicion you likely performed an exploratory study.

"Non-important" variables, i.e. variables that do not explain much variance in the outcome variable, will only exert undue influence on your data if you have too many predictors relative to your sample size (overfitting) or if there are predictors that are highly correlated (collinear). Ideally you avoid these situations by performing an exploratory study.

One way to check for overfitting/unstable model problems is by exploring a "reduced model" that includes "significant" terms from the main model, only. Importantly, this reduced model analysis should be referred to as a post-hoc control analysis aiding interpretation. Conclusions should solely be based on the pre-specified model.

  • $\begingroup$ thanks, this is very useful. With respect to your last point, once you fitted the full and this reduced model (with only "significant" terms), what would you base your comparison on? $\endgroup$ – Tilen Mar 15 '17 at 17:13
  • $\begingroup$ My reality (in biology) is that there is often a sort of combination of these two points. We are often trying to infer relationships in nature, but at the same time trying to be able to predict (albeit in an exploratory way). $\endgroup$ – Tilen Mar 15 '17 at 17:15
  • $\begingroup$ Regarding your first question: What do you mean with comparison? As stated above all inferences must be based on the pre-specified model. Regarding the second question: Mixing exploration and inference is also called "guesstimation" and will lead you towards unreliable results. All depends on your aims. You have to get clarity for yourself whether you want to explore, make inferences or make predictions. $\endgroup$ – mzunhammer Mar 16 '17 at 15:15
  • $\begingroup$ You said that "one way to check for overfitting/unstable model problems is by exploring a "reduced model" that includes "significant" terms from the main model, only." So how do you explore this reduced model? I thought you meant to compare it with the full one somehow (that's why I wrote comparison). With respect to second point, I'm not sure I fully understand the difference between exploring relationships vs. making inferences about these relationships... $\endgroup$ – Tilen Mar 16 '17 at 16:59
  • $\begingroup$ Creating the reduced model is the exploratory step. You should check if any factor shows up as significant in the main analysis, but not so in the reduced model. If this happens you should check why. Regarding the exploratory vs inferential study: Have a look at the references cited above. $\endgroup$ – mzunhammer Mar 17 '17 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.