Despite reading several online references, including the full Wikipedia article on "ANOVA", I'm still confused at the recommended process taken to build the most statistically significant linear or logistic model for a single dependent response variable (in my case, it may be either continuous or categorical) with a large number of possible independent variables (which also may be either continuous or categorical).
In my head, I can see at least 2 possible approaches:
- Additive approach. Start with empty model and, at each iteration, add the next most significant (via F-test) independent variable or interation (after exhaustively testing all remaining variables/interactions). Proceed until no more statistically significant additions can be found.
- Subtractive approach. Start with a complete model including all variables and all their interactions, and iteratively remove the least significant term (via F-test). Continue until all remaining terms are significant.
Is there another approach I'm missing? What is considered the best approach for model building by contemporary literature? Sorry if I'm asking amateur questions here; feel free to answer with existing online references if that can make any answers more concise.
I will need to implement the solution programmatically, so I need to know the process moreso than anything else.