Despite reading several online references, including the full Wikipedia article on "ANOVA", I'm still confused at the recommended process taken to build the most statistically significant linear or logistic model for a single dependent response variable (in my case, it may be either continuous or categorical) with a large number of possible independent variables (which also may be either continuous or categorical).

In my head, I can see at least 2 possible approaches:

  1. Additive approach. Start with empty model and, at each iteration, add the next most significant (via F-test) independent variable or interation (after exhaustively testing all remaining variables/interactions). Proceed until no more statistically significant additions can be found.
  2. Subtractive approach. Start with a complete model including all variables and all their interactions, and iteratively remove the least significant term (via F-test). Continue until all remaining terms are significant.

Is there another approach I'm missing? What is considered the best approach for model building by contemporary literature? Sorry if I'm asking amateur questions here; feel free to answer with existing online references if that can make any answers more concise.

I will need to implement the solution programmatically, so I need to know the process moreso than anything else.

  • 3
    $\begingroup$ How can a model in itself be "the most statistically significant" if you are using ad hoc hypotheses all the way up and down your model selection strategy to accept or reject regressors all higgledy piggledy? What is the grand null hypothesis? Stepwise model selection is not a valid means of testing hypotheses. $\endgroup$ – AdamO Jan 4 at 18:47
  • $\begingroup$ @AdamO I acknowledge my amateur knowledge. Can you post an answer on how you'd recommend to combine model building / selection with hypothesis testing? $\endgroup$ – Special Sauce Jan 4 at 18:51
  • $\begingroup$ I assume the grand null hypothesis is that none of the independent variables are statistically significant. In the additive approach, I supposed I would test the added term with the extra degree of freedom was more significant than the prior model without the new term. $\endgroup$ – Special Sauce Jan 4 at 18:57

There are many model building and model selection techniques you could use to select the "best" model -- too many to name here, so I'll focus on one that I personally use quite often and think is quite good. Before we discuss it, you should understand that model building generally tries to select the most parsimonious model that still accurately reflects the characteristics of your data as minimizing the number of variables makes your models easier to interpret, makes your model less prone to overfitting, and increases the chances that your model will be numerically stable, among other nice properties.

With that being said, I like to use a technique called "Purposeful Selection" Model Building or Purposeful Selection of Covariates because it usually produces good, parsimonious model properties. The technique essentially consists of the following steps that I have very briefly summarized below. See the references below for details of each step:

  1. Conduct a univariable analysis of each independent variable and identity a set of candidate variables to be included in your model.
  2. Fit your model containing all candidates variables and assess the importance of each variable using your model statistics. Eliminate any variables that do not achieve your desired level of statistical significance.
  3. Examine your "reduced model" from step 2 for any drastic changes in coefficients from the larger model with all candidates variables. Add back any variables to your model that result in large changes in magnitude to your coefficients.
  4. Add back into the model each variable not selected as a candidate in step 1. Keep any variables that become significant at this step. Call this model the main effects model.
  5. Check for interaction terms and add in those interactions that are significant and make sense into your main effects model.
  6. Assess your new model for fit, violations, of assumptions, etc.

At the end of this step, you should have a final model. Keep in mind many of these steps are cyclical and repeated. You can read more about the details of this procedure in Bursac et al. (2008).

Keep in mind that this is not the only approach. There are other "machine-learning" techniques and statistical techniques that can be used for model building as well. You might want to consider reading the paper by Harrell et al. (1996) as they discuss common problems with the model selection process, evaluating assumptions and adequacy, and measuring and reducing errors.

  • $\begingroup$ Thanks for posting a simple meta-process that makes sense. I agree that parsimony is a very logical objective of the model building process. Would you be able to elaborate a bit more on Step 5 above? How do you determine the significance of potential additional interaction terms with respect to the existing main effects model? Do you recommend to consider all interactions exhaustively? Or only interactions of terms already in the main effects model? $\endgroup$ – Special Sauce Jan 4 at 19:25
  • $\begingroup$ @SpecialSauce If interpretability is important for you, I'd only consider adding in interactions that appear as main effects. If you check out the paper by Bursac, they provide a very detailed overview of that step too with some alternative considerations. But I personally an of the opinion that you can't have an interaction in a model unless the main effects are also present. $\endgroup$ – StatsStudent Jan 4 at 19:28
  • 1
    $\begingroup$ Thanks for providing the Bursac paper reference; already reading it and the authors are illuminating on many of my concerns, while also providing a straight-forward process (even with a flow chart). $\endgroup$ – Special Sauce Jan 4 at 19:39
  • $\begingroup$ @SpecialSauce, you are welcome. I forgot to mention, that you'll find some more details on this in Applied Logistic Regression, 3d. by Hosmer, Lemeshow, and Sturdivant in the chapter on model selection and model building techniques. They are biostatisticians, so they naturally make lots of references to "clinical importance" but if you replace the words "clinical importance" with "importance to your field" then the methods apply to whatever field you are studying. They also have a few worked examples of applying purposeful selection, if I recall correctly. $\endgroup$ – StatsStudent Jan 4 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.