Do all interactions terms need their individual terms in regression model?

Question

I am actually reviewing a manuscript where the authors compare 5-6 logit regression models with AIC. However, some of the models have interaction terms without including the individual covariate terms. Does it ever make sense to do this?

For example (not specific to logit models):

M1: Y = X1 + X2 + X1*X2
M2: Y = X1 + X2
M3: Y = X1 + X1*X2 (missing X2)
M4: Y = X2 + X1*X2 (missing X1)
M5: Y = X1*X2 (missing X1 & X2)

I've always been under the impression that if you have the interaction term X1*X2 you also need X1 + X2. Therefore, models 1 and 2 would be fine but models 3-5 would be problematic (even if AIC is lower). Is this correct? Is it a rule or more of a guideline? Does anyone have a good reference that explains the reasoning behind this? I just want to make sure I don't miscommunicate anything important in the review.

Answer 1

Most of the time this is a bad idea - the main reason is that it no longer makes the model invariant to location shifts. For example, suppose you have a single outcome $y_i$ and two predictors $x_i$ and $z_i$ and specify the model:

$$ y_i = \beta_0 + \beta_1 x_{i} z_i + \varepsilon $$

If you were to center the predictors by their means, $x_i z_i$ becomes

$$ (x_i - \overline{x})(z_i - \overline{z}) = x_i z_i - x_{i} \overline{z} - z_{i} \overline{x} + \overline{x} \overline{z}$$

So, you can see that the main effects have been reintroduced into the model.

I've given a heuristic argument here, but this does present a practical issue. As noted in Faraway(2005) on page 114, an additive change in scale changes the model inference when the main effects are left out of the model, whereas this does not happen when the lower order terms are included. It is normally undesirable to have arbitrary things like a location shift cause a fundamental change in the statistical inference (and therefore the conclusions of your inquiry), as can happen when you include polynomial terms or interactions in a model without the lower order effects.

Note: There may be special circumstances where you would only want to include the interaction, if the $x_i z_i$ has some particular substantive meaning or if you only observe the product and not the individual variables $x_i, z_i$. But, in that case, one may as well think of the predictor $a_i = x_i z_i$ and proceed with the model

$$ y_i = \alpha_0 + \alpha_1 a_i + \varepsilon_i $$

rather than thinking of $a_i$ as an interaction term.

Answer 2

All the answers so far seem to miss a very basic point: the functional form you choose should be flexible enough to capture the features that are scientifically relevant. Models 2-5 impose zero coefficients on some terms without scientific justification. And even if scientifically justified, Model 1 remains appealing because you might as well test for the zero coefficients rather than impose them.

The key is understanding what the restrictions mean. The typical admonition to avoid Models 3-5 is because in most applications the assumptions they impose are scientifically implausible. Model 3 assumes X2 only influences the slope dY/dX1 but not the level. Model 4 assumes X1 only influences the slope dY/dX2 but not the level. And Model 5 assumes neither X1 nor X2 affects the level, but only dY/dX1 or dY/dX2. In most applications these assumptions don't seem reasonable. Model 2 also imposes a zero coefficient but still has some merit. It gives the best linear approximation to the data, which in many cases satisfies the scientific goal.

Answer 3

+1 to @Macro. Let me bring out what I think is a similar point that concerns when you have categorical predictors. A lot can depend on how they are coded. For example, reference cell (aka, 'dummy') coding uses 0 & 1, whereas effect coding uses -1, 0 & 1. Consider a simple case with two factors with two levels each, then $x_1x_2$ could be [0, 0, 0, 1] or [1, -1, -1, 1], depending on the coding scheme used. I believe that it is possible to have a situation where only the interaction is 'significant' with one coding scheme, but all terms are 'significant' using the other scheme. This implies that meaningful interpretive decisions would be made based on an arbitrary coding decision that, in fact, your software may have made for you without your knowledge. I recognize that this is a small point, but it's just one more reason that it's typically not a good idea to retain only the interaction (and also not to select a subset of predictors based on p-values, of course).

Answer 4

Since you are reviewing a paper you might suggest that the authors discuss the issue of model hierarchy and justify their departure from it.

Here are some references:

1. Nelder JA. The selection of terms in response-surface models—how strong is the weak-heredity principle? The American Statistician. 1998;52:315–8. http://www.jstor.org/pss/2685433. Accessed 10 June 2010.

2. Peixoto JL. Hierarchical variable selection in polynomial regression models. The American Statistician. 1987;41:311–3. http://www.jstor.org/pss/2684752. Accessed 10 June 2010.

3. Peixoto JL. A property of well-formulated polynomial regression models. The American Statistician. 1990;44:26–30. http://www.jstor.org/pss/2684952. Accessed 10 June 2010.

I usually follow hierarchy but depart from it in some situations. For example, if you are testing tire wear versus mileage at several different speeds, your model might look like:

tread depth = intercept + mileage + mileage*speed

but it would not make physical sense to include a main effect of speed because the tire does not know what the speed will be at zero miles.

(On the other hand, you might still want to test for a speed effect because it might indicate that "break-in" effects differ at different speeds. On the other other hand, an even better way to handle break-in would be to get data at zero and at very low mileage and then test for non-linearity. Note that removing the intercept term can be thought of as a special case of violating hierarchy.)

I'll also reiterate what someone said above because it's very important: The authors need to make sure they know whether their software is centering the data. The tire model above becomes physically nonsensical if the software replaces mileage with (mileage - mean of mileage).

The same sorts of things are relevant in pharmaceutical stability studies (mentioned tangentially in "Stability Models for Sequential Storage", Emil M. Friedman and Sam C. Shum, AAPS PharmSciTech, Vol. 12, No. 1, March 2011, DOI: 10.1208/s12249-010-9558-x).

Answer 5

I have had a real case that illustrates this. In the data, one of the variables represented group with 0-control and 1-treatment. The other predictor represented time period with 0-before treatment and 1-after treatment. The interaction was the main parameter of interest measuring the effect of the treatment, the difference after the treatment in the treatment group above any effect of time measured in the control group. The main effect from group measured the difference in the 2 groups before any treatment, so it could easily be 0 (in a randomized experiment it should be 0, this one was not). The 2nd main effect measures the difference between the before and after time periods in the control group where there was no treatment, so this also makes sense that it could be 0 while the interaction term is non-zero. Of course this depends on how things were coded and a different coding would change the meanings and whether or not the interaction makes sense without the main effects. So it only makes sense to fit the interaction without the main effects in specific cases.

Answer 6

I agree with Peter. I think the rule is folklore. Why could we conceive of a situation where two variables would affect the model only because of an interaction. An analogy in chemistry is that two chemicals are totally inert by themselves but cause an explosion when mixed together. Mathematical/statistical niceties like invariance have nothing to do with a real problem with real data. I just think that when there are a lot of variables to consider there is an awful lot of testing to do if you are going to look at all main effects and most if not all first order interactions. We also almost never look at second order interactions even in small experiments with only a handful of variables. The thinking is that the higher the order of interaction the less likely it is that there is a real effect. So don't look at first or second order interactions if the main effect isn't there. A good rule perhaps but to follow it religiously means overlooking the exceptions and your problem may be an exception.

Answer 7

[trying to answer a part of the original question which seems left uncovered in most answers: "should AIC, as a model selection criterion be trusted?"]

AIC should be used more as a guideline, than a rule that should be taken as gospel.

The effectiveness of AIC (or BIC or any similar 'simple' criterion for model selection) highly depends on the learning algorithm, and the problem.

Think of it this way: the goal of the complexity (number of factors) term in the AIC formula is simple: to avoid selecting models which over-fit. But the simplicity of AIC very often fails to capture the real complexity of the problem itself. This is why there are other practical techniques to avoid over-fitting: for example, cross-validation or adding a regularization term.

When I use online SGD (stochastic gradient descent) to do linear regression on a data-set with a very large number of inputs, I find AIC to be a terrible predictor of model quality because it excessively penalizes complex models with large number of terms. There are many real life situations where each term has a tiny effect, but together, a large number of them gives strong statistical evidence of an outcome. AIC and BIC model-selection criteria would reject these models and prefer the simpler ones, even though the more complex ones are superior.

In the end, it is the generalization error (roughly: out of sample performance) that counts. AIC can give you some hint of model quality in some relatively simple situations. Just be careful and remember that real life is more often than not, more complex than a simple formula.