recommended approach for building an ANOVA model

Question

Despite reading several online references, including the full Wikipedia article on "ANOVA", I'm still confused at the recommended process taken to build the most statistically significant linear or logistic model for a single dependent response variable (in my case, it may be either continuous or categorical) with a large number of possible independent variables (which also may be either continuous or categorical).

In my head, I can see at least 2 possible approaches:

1. Additive approach. Start with empty model and, at each iteration, add the next most significant (via F-test) independent variable or interation (after exhaustively testing all remaining variables/interactions). Proceed until no more statistically significant additions can be found.
2. Subtractive approach. Start with a complete model including all variables and all their interactions, and iteratively remove the least significant term (via F-test). Continue until all remaining terms are significant.

Is there another approach I'm missing? What is considered the best approach for model building by contemporary literature? Sorry if I'm asking amateur questions here; feel free to answer with existing online references if that can make any answers more concise.

I will need to implement the solution programmatically, so I need to know the process moreso than anything else.

Answer 1

There are many model building and model selection techniques you could use to select the "best" model -- too many to name here, so I'll focus on one that I personally use quite often and think is quite good. Before we discuss it, you should understand that model building generally tries to select the most parsimonious model that still accurately reflects the characteristics of your data as minimizing the number of variables makes your models easier to interpret, makes your model less prone to overfitting, and increases the chances that your model will be numerically stable, among other nice properties.

With that being said, I like to use a technique called "Purposeful Selection" Model Building or Purposeful Selection of Covariates because it usually produces good, parsimonious model properties. The technique essentially consists of the following steps that I have very briefly summarized below. See the references below for details of each step:

1. Conduct a univariable analysis of each independent variable and identity a set of candidate variables to be included in your model.
2. Fit your model containing all candidates variables and assess the importance of each variable using your model statistics. Eliminate any variables that do not achieve your desired level of statistical significance.
3. Examine your "reduced model" from step 2 for any drastic changes in coefficients from the larger model with all candidates variables. Add back any variables to your model that result in large changes in magnitude to your coefficients.
4. Add back into the model each variable not selected as a candidate in step 1. Keep any variables that become significant at this step. Call this model the main effects model.
5. Check for interaction terms and add in those interactions that are significant and make sense into your main effects model.
6. Assess your new model for fit, violations, of assumptions, etc.

At the end of this step, you should have a final model. Keep in mind many of these steps are cyclical and repeated. You can read more about the details of this procedure in Bursac et al. (2008).

Keep in mind that this is not the only approach. There are other "machine-learning" techniques and statistical techniques that can be used for model building as well. You might want to consider reading the paper by Harrell et al. (1996) as they discuss common problems with the model selection process, evaluating assumptions and adequacy, and measuring and reducing errors.