I am actually reviewing a manuscript where the authors compare 5-6 logit regression models with AIC. However, some of the models have interaction terms without including the individual covariate terms. Does it ever make sense to do this?

For example (not specific to logit models):

M1: Y = X1 + X2 + X1*X2
M2: Y = X1 + X2
M3: Y = X1 + X1*X2 (missing X2)
M4: Y = X2 + X1*X2 (missing X1)
M5: Y = X1*X2 (missing X1 & X2)

I've always been under the impression that if you have the interaction term X1*X2 you also need X1 + X2. Therefore, models 1 and 2 would be fine but models 3-5 would be problematic (even if AIC is lower). Is this correct? Is it a rule or more of a guideline? Does anyone have a good reference that explains the reasoning behind this? I just want to make sure I don't miscommunicate anything important in the review.

Thanks for any thoughts, Dan

  • 8
    $\begingroup$ +1, I think this is a really good question. You may also want to check out this earlier question that covers much of the same territory. The answers there are really excellent as well. $\endgroup$ – gung - Reinstate Monica May 4 '12 at 3:17
  • $\begingroup$ Lots of good answers already. There was a paper by Rindskopf on some cases where you do not need the main effects. (Also see this one) $\endgroup$ – Peter Flom May 4 '12 at 11:49
  • 4
    $\begingroup$ AFAIK: in R's lm(), : is for interactions, as in A:B. And * is for both main effects and interactions, so A*B = A+B+A:B. So if(!) the authors of the paper follow this notation, I don't think any of the models are missing th emain effects? $\endgroup$ – Zhubarb Aug 18 '14 at 13:35
  • $\begingroup$ Also, the same logic as in the current answers applies to higher order interactions (e.g. you need all the 2 way interactions if you include a 3 way) $\endgroup$ – Peter Flom Oct 18 '19 at 12:58

Most of the time this is a bad idea - the main reason is that it no longer makes the model invariant to location shifts. For example, suppose you have a single outcome $y_i$ and two predictors $x_i$ and $z_i$ and specify the model:

$$ y_i = \beta_0 + \beta_1 x_{i} z_i + \varepsilon $$

If you were to center the predictors by their means, $x_i z_i$ becomes

$$ (x_i - \overline{x})(z_i - \overline{z}) = x_i z_i - x_{i} \overline{z} - z_{i} \overline{x} + \overline{x} \overline{z}$$

So, you can see that the main effects have been reintroduced into the model.

I've given a heuristic argument here, but this does present a practical issue. As noted in Faraway(2005) on page 114, an additive change in scale changes the model inference when the main effects are left out of the model, whereas this does not happen when the lower order terms are included. It is normally undesirable to have arbitrary things like a location shift cause a fundamental change in the statistical inference (and therefore the conclusions of your inquiry), as can happen when you include polynomial terms or interactions in a model without the lower order effects.

Note: There may be special circumstances where you would only want to include the interaction, if the $x_i z_i$ has some particular substantive meaning or if you only observe the product and not the individual variables $x_i, z_i$. But, in that case, one may as well think of the predictor $a_i = x_i z_i$ and proceed with the model

$$ y_i = \alpha_0 + \alpha_1 a_i + \varepsilon_i $$

rather than thinking of $a_i$ as an interaction term.

  • $\begingroup$ additive change in scale changes the inference (the t -statistics) for all but the highest order terms when any lower order terms are left out of the model Additive change of predictors generally changes t of their main effects (lower order terms) even in a full model. It is overall fit (R^2) that is preserved (but is not preserved under additive change in a model with some main effects dropped). Is that what you wanted to say? $\endgroup$ – ttnphns May 6 '12 at 14:32
  • $\begingroup$ Yes, that's right @ttnphns - thanks for pointing that out - I've modified my answer a bit to reflect this. $\endgroup$ – Macro May 8 '12 at 2:51

All the answers so far seem to miss a very basic point: the functional form you choose should be flexible enough to capture the features that are scientifically relevant. Models 2-5 impose zero coefficients on some terms without scientific justification. And even if scientifically justified, Model 1 remains appealing because you might as well test for the zero coefficients rather than impose them.

The key is understanding what the restrictions mean. The typical admonition to avoid Models 3-5 is because in most applications the assumptions they impose are scientifically implausible. Model 3 assumes X2 only influences the slope dY/dX1 but not the level. Model 4 assumes X1 only influences the slope dY/dX2 but not the level. And Model 5 assumes neither X1 nor X2 affects the level, but only dY/dX1 or dY/dX2. In most applications these assumptions don't seem reasonable. Model 2 also imposes a zero coefficient but still has some merit. It gives the best linear approximation to the data, which in many cases satisfies the scientific goal.

  • 5
    $\begingroup$ (+1) This is all true, but the original poster seemed to be describing a situation where the authors were trying to do model selection, and some of their candidate models were ones that did not include interactions - so their motivation was guided by AIC rather than by something substantive (which is always a dangerous thing to do, but apparently they've done it). When you're guided by something substantive, then the model structure should be dictated by that. But, when you're guided by a statistical criteria, leaving out main effects can have bad properties, as I've indicated in my answer. $\endgroup$ – Macro May 4 '12 at 16:40

+1 to @Macro. Let me bring out what I think is a similar point that concerns when you have categorical predictors. A lot can depend on how they are coded. For example, reference cell (aka, 'dummy') coding uses 0 & 1, whereas effect coding uses -1, 0 & 1. Consider a simple case with two factors with two levels each, then $x_1x_2$ could be [0, 0, 0, 1] or [1, -1, -1, 1], depending on the coding scheme used. I believe that it is possible to have a situation where only the interaction is 'significant' with one coding scheme, but all terms are 'significant' using the other scheme. This implies that meaningful interpretive decisions would be made based on an arbitrary coding decision that, in fact, your software may have made for you without your knowledge. I recognize that this is a small point, but it's just one more reason that it's typically not a good idea to retain only the interaction (and also not to select a subset of predictors based on p-values, of course).

  • 1
    $\begingroup$ Significance testing for categorical main effects is not any less invariant. A group may be significantly different from the reference group under treatment coding but not from the "grand mean" effect under contrast coding. $\endgroup$ – probabilityislogic May 9 '12 at 2:19

Since you are reviewing a paper you might suggest that the authors discuss the issue of model hierarchy and justify their departure from it.

Here are some references:

  1. Nelder JA. The selection of terms in response-surface models—how strong is the weak-heredity principle? The American Statistician. 1998;52:315–8. http://www.jstor.org/pss/2685433. Accessed 10 June 2010.

  2. Peixoto JL. Hierarchical variable selection in polynomial regression models. The American Statistician. 1987;41:311–3. http://www.jstor.org/pss/2684752. Accessed 10 June 2010.

  3. Peixoto JL. A property of well-formulated polynomial regression models. The American Statistician. 1990;44:26–30. http://www.jstor.org/pss/2684952. Accessed 10 June 2010.

I usually follow hierarchy but depart from it in some situations. For example, if you are testing tire wear versus mileage at several different speeds, your model might look like:

tread depth = intercept + mileage + mileage*speed

but it would not make physical sense to include a main effect of speed because the tire does not know what the speed will be at zero miles.

(On the other hand, you might still want to test for a speed effect because it might indicate that "break-in" effects differ at different speeds. On the other other hand, an even better way to handle break-in would be to get data at zero and at very low mileage and then test for non-linearity. Note that removing the intercept term can be thought of as a special case of violating hierarchy.)

I'll also reiterate what someone said above because it's very important: The authors need to make sure they know whether their software is centering the data. The tire model above becomes physically nonsensical if the software replaces mileage with (mileage - mean of mileage).

The same sorts of things are relevant in pharmaceutical stability studies (mentioned tangentially in "Stability Models for Sequential Storage", Emil M. Friedman and Sam C. Shum, AAPS PharmSciTech, Vol. 12, No. 1, March 2011, DOI: 10.1208/s12249-010-9558-x).

  • 1
    $\begingroup$ thank you, this is a great answer and will help me explain it to people who are not statistically savvy. $\endgroup$ – djhocking May 9 '12 at 12:54
  • 1
    $\begingroup$ +1 I wish it was possible to merge answers on SO. This with the accepted answer above forms the perfect answer. $\endgroup$ – Zhubarb Apr 30 '15 at 8:06

I have had a real case that illustrates this. In the data, one of the variables represented group with 0-control and 1-treatment. The other predictor represented time period with 0-before treatment and 1-after treatment. The interaction was the main parameter of interest measuring the effect of the treatment, the difference after the treatment in the treatment group above any effect of time measured in the control group. The main effect from group measured the difference in the 2 groups before any treatment, so it could easily be 0 (in a randomized experiment it should be 0, this one was not). The 2nd main effect measures the difference between the before and after time periods in the control group where there was no treatment, so this also makes sense that it could be 0 while the interaction term is non-zero. Of course this depends on how things were coded and a different coding would change the meanings and whether or not the interaction makes sense without the main effects. So it only makes sense to fit the interaction without the main effects in specific cases.

  • $\begingroup$ So do you mean it all depends on the objectives of your study OR based on your parameters? $\endgroup$ – Ben Sep 4 '14 at 12:44
  • 1
    $\begingroup$ @Ben, it can depend on both how you parameterize your variables (in my example switching the 0/1 to 1/0 for either variable would change the interpretation) and what questions you are trying to answer and what assumptions you are willing to make. $\endgroup$ – Greg Snow Sep 4 '14 at 17:33
  • $\begingroup$ thanks for your reply. I have two ways interaction and want to interact that with a dummy year. I only want to assess the year 2008 (regulation year) on the two ways interaction. so is it ok to use $$ Y= B_0+B_1 X+B_2 Z+B_3X*Z*2008+yeardummies $$ X & Z are continuous variables, Z is the regulation rating. the year 2008 is scored 1 and 0 for other years. so it is like if i only take the observations of the year 2008 without interaction. I read about the weak and strong heredity principle, but didn't clearly understand $\endgroup$ – Ben Sep 7 '14 at 15:26
  • 1
    $\begingroup$ @Ben, It is certainly possible to fit the above model which basically says you think there is (or may be) an interaction in 2008, but not in any other year. If you have a justification for this, then I think the model is fine. But it is an unusual enough assumption that you will probably need to justify this to any audience. $\endgroup$ – Greg Snow Sep 8 '14 at 16:00
  • $\begingroup$ thanks a lot that was helpful. Is it ok to compare the output of the interaction with $$B1X$$ although $$B1X$$ is for the whole period and the interaction is only for the year 2008 $\endgroup$ – Ben Sep 24 '14 at 12:57

I agree with Peter. I think the rule is folklore. Why could we conceive of a situation where two variables would affect the model only because of an interaction. An analogy in chemistry is that two chemicals are totally inert by themselves but cause an explosion when mixed together. Mathematical/statistical niceties like invariance have nothing to do with a real problem with real data. I just think that when there are a lot of variables to consider there is an awful lot of testing to do if you are going to look at all main effects and most if not all first order interactions. We also almost never look at second order interactions even in small experiments with only a handful of variables. The thinking is that the higher the order of interaction the less likely it is that there is a real effect. So don't look at first or second order interactions if the main effect isn't there. A good rule perhaps but to follow it religiously means overlooking the exceptions and your problem may be an exception.

  • 8
    $\begingroup$ Re: "Mathematical/statistical niceties like invariance have nothing to do with a real problem with real data" - it does have to do with a real problem with real data when your $p$-values, and therefore your statistical inference (and therefore your "real world" decision about the importance of a predictor), can depend on something as arbitrary as the decision to center your predictors. $\endgroup$ – Macro May 4 '12 at 19:59
  • 1
    $\begingroup$ I probably misspoke saying that invariance has no relevance in the real world. My intended point was that some mathematical results may not be relevant in a particular practical problem. As an example least squares estimates are maximum likelihood under normal error assumptions and by the Gauss Markov theorem are minimum variance unbiased under weaker conditions, but I wouldn't use it when there are outliers in the data. By the same token should a property like invariance rule out including an interaction when it makes sense say medically that it would occur without the main effects? $\endgroup$ – Michael R. Chernick May 5 '12 at 18:09

[trying to answer a part of the original question which seems left uncovered in most answers: "should AIC, as a model selection criterion be trusted?"]

AIC should be used more as a guideline, than a rule that should be taken as gospel.

The effectiveness of AIC (or BIC or any similar 'simple' criterion for model selection) highly depends on the learning algorithm, and the problem.

Think of it this way: the goal of the complexity (number of factors) term in the AIC formula is simple: to avoid selecting models which over-fit. But the simplicity of AIC very often fails to capture the real complexity of the problem itself. This is why there are other practical techniques to avoid over-fitting: for example, cross-validation or adding a regularization term.

When I use online SGD (stochastic gradient descent) to do linear regression on a data-set with a very large number of inputs, I find AIC to be a terrible predictor of model quality because it excessively penalizes complex models with large number of terms. There are many real life situations where each term has a tiny effect, but together, a large number of them gives strong statistical evidence of an outcome. AIC and BIC model-selection criteria would reject these models and prefer the simpler ones, even though the more complex ones are superior.

In the end, it is the generalization error (roughly: out of sample performance) that counts. AIC can give you some hint of model quality in some relatively simple situations. Just be careful and remember that real life is more often than not, more complex than a simple formula.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.