Does anyone have any good resources for building explanatory regression models? Using the distinction between explanatory vs. predictive models described in Shmueli (2010) (text available here), I'm wanting to building an explanatory model because I'm interested in examining how several variables specially affect the outcome variable. I'm asking this question here because although I've found advice for building predictive models (e.g., by Gelman and Hill (2006): http://www.stat.columbia.edu/~gelman/arm/), and I'm thinking this process might be different for explanatory models.

To put this question in more context, I have four variables of interest. Plus, I'm expecting these variables to have different effects for males and females, and I'd like to also test some additional covariates that may or may not be important. What I'd like is a guide on how to best build models with these variables.

For example, in an ANOVA approach, I think the convention is to just add the 5 variables (4 I care about and gender) all at once and test the highest order interaction possible (i.e., fit a complex model right off the bat). Does this make sense to apply to a multiple regression framework? If so, when would I add the covariates? These are the types of questions I'm trying to resolve.


Gelman, A. and Hill, J., 2006. Data analysis using regression and multilevel/hierarchical models. Cambridge university press.

Shmueli, G., 2010. To explain or to predict?. Statistical science, 25(3), pp.289-310, doi:10.1214/10-STS330.

  • $\begingroup$ Gelman and Hill IS a book about explanatory modeling, not prediction. $\endgroup$
    – Eoin
    Sep 16, 2022 at 7:21

1 Answer 1


One major difference between prediction and explanation is that it's hard to interpret interactions of order higher than two, even if they may be important for a better prediction. In addition, you may wish to restrict the model pool to hierarchical models only (if A*B is included, then both main effects, A and B, should be included).

Other than that, you can use any method you want (stepwise, forward selection, backward elimination, AIC, etc). I find that combining backward elimination with AIC works fine: you drop the term with the largest p-value until you find the model with the lowest AIC. To interpret AIC values correctly, look up how Akaike weights are defined and calculate them for two or more competing models.

Another possibility is to use CART which is also interpretable if you limit yourself to a single tree. That way, you can essentially consider higher order interactions without turning it into a black box.

  • $\begingroup$ Great, thanks for the post! So just to be clear though, you would start with a high level interaction (in my case a 5 way interaction) and then use backward elimination with aid of the AIC? $\endgroup$ Nov 29, 2016 at 16:18
  • $\begingroup$ No, I wouldn't go above 2nd order interaction. Even if 3rd or higher order interaction is significant, you'll have a hard time explaining what it means. $\endgroup$
    – Nik Tuzov
    Nov 30, 2016 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.