Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am actually reviewing a manuscript where the authors compare 5-6 logit regression models with AIC. However, some of the models have interaction terms without including the individual covariate terms. Does it ever make sense to do this?

For example (not specific to logit models):

M1: Y = X1 + X2 + X1*X2
M2: Y = X1 + X2
M3: Y = X1 + X1*X2 (missing X2)
M4: Y = X2 + X1*X2 (missing X1)
M5: Y = X1*X2 (missing X1 & X2)

I've always been under the impression that if you have the interaction term X1*X2 you also need X1 + X2. Therefore, models 1 and 2 would be fine but models 3-5 would be problematic (even if AIC is lower). Is this correct? Is it a rule or more of a guideline? Does anyone have a good reference that explains the reasoning behind this? I just want to make sure I don't miscommunicate anything important in the review.

Thanks for any thoughts, Dan

share|improve this question
5  
+1, I think this is a really good question. You may also want to check out this earlier question that covers much of the same territory. The answers there are really excellent as well. –  gung May 4 '12 at 3:17
add comment

8 Answers

up vote 22 down vote accepted

Most of the time this is a bad idea - the main reason is that it no longer makes the model invariant to location shifts. For example, suppose you have a single outcome $y_i$ and two predictors $x_i$ and $z_i$ and specify the model:

$$ y_i = \beta_0 + \beta_1 x_{i} z_i + \varepsilon $$

If you were to center the predictors by their means, $x_i z_i$ becomes

$$ (x_i - \overline{x})(z_i - \overline{z}) = x_i z_i - x_{i} \overline{z} - z_{i} \overline{x} + \overline{x} \overline{z}$$

So, you can see that the main effects have been reintroduced into the model.

I've given a heuristic argument here, but this does present a practical issue. As noted in Faraway(2005) on page 114, an additive change in scale changes the model inference when the main effects are left out of the model, whereas this does not happen when the lower order terms are included. It is normally undesirable to have arbitrary things like a location shift cause a fundamental change in the statistical inference (and therefore the conclusions of your inquiry), as can happen when you include polynomial terms or interactions in a model without the lower order effects.

Note: There may be special circumstances where you would only want to include the interaction, if the $x_i z_i$ has some particular substantive meaning or if you only observe the product and not the individual variables $x_i, z_i$. But, in that case, one may as well think of the predictor $a_i = x_i z_i$ and proceed with the model

$$ y_i = \alpha_0 + \alpha_1 a_i + \varepsilon_i $$

rather than thinking of $a_i$ as an interaction term.

share|improve this answer
    
additive change in scale changes the inference (the t -statistics) for all but the highest order terms when any lower order terms are left out of the model Additive change of predictors generally changes t of their main effects (lower order terms) even in a full model. It is overall fit (R^2) that is preserved (but is not preserved under additive change in a model with some main effects dropped). Is that what you wanted to say? –  ttnphns May 6 '12 at 14:32
    
Yes, that's right @ttnphns - thanks for pointing that out - I've modified my answer a bit to reflect this. –  Macro May 8 '12 at 2:51
add comment

All the answers so far seem to miss a very basic point: the functional form you choose should be flexible enough to capture the features that are scientifically relevant. Models 2-5 impose zero coefficients on some terms without scientific justification. And even if scientifically justified, Model 1 remains appealing because you might as well test for the zero coefficients rather than impose them.

The key is understanding what the restrictions mean. The typical admonition to avoid Models 3-5 is because in most applications the assumptions they impose are scientifically implausible. Model 3 assumes X2 only influences the slope dY/dX1 but not the level. Model 4 assumes X1 only influences the slope dY/dX2 but not the level. And Model 5 assumes neither X1 nor X2 affects the level, but only dY/dX1 or dY/dX2. In most applications these assumptions don't seem reasonable. Model 2 also imposes a zero coefficient but still has some merit. It gives the best linear approximation to the data, which in many cases satisfies the scientific goal.

share|improve this answer
3  
(+1) This is all true, but the original poster seemed to be describing a situation where the authors were trying to do model selection, and some of their candidate models were ones that did not include interactions - so their motivation was guided by AIC rather than by something substantive (which is always a dangerous thing to do, but apparently they've done it). When you're guided by something substantive, then the model structure should be dictated by that. But, when you're guided by a statistical criteria, leaving out main effects can have bad properties, as I've indicated in my answer. –  Macro May 4 '12 at 16:40
add comment

+1 to @Macro. Let me bring out what I think is a similar point that concerns when you have categorical predictors. A lot can depend on how they are coded. For example, reference cell (aka, 'dummy') coding uses 0 & 1, whereas effect coding uses -1, 0 & 1. Consider a simple case with two factors with two levels each, then $x_1x_2$ could be [0, 0, 0, 1] or [1, -1, -1, 1], depending on the coding scheme used. I believe that it is possible to have a situation where only the interaction is 'significant' with one coding scheme, but all terms are 'significant' using the other scheme. This implies that meaningful interpretive decisions would be made based on an arbitrary coding decision that, in fact, your software may have made for you without your knowledge. I recognize that this is a small point, but it's just one more reason that it's typically not a good idea to retain only the interaction (and also not to select a subset of predictors based on p-values, of course).

share|improve this answer
1  
Significance testing for categorical main effects is not any less invariant. A group may be significantly different from the reference group under treatment coding but not from the "grand mean" effect under contrast coding. –  probabilityislogic May 9 '12 at 2:19
add comment

Since you are reviewing a paper you might suggest that the authors discuss the issue of model hierarchy and justify their departure from it.

Here are some references:

  1. Nelder JA. The selection of terms in response-surface models—how strong is the weak-heredity principle? The American Statistician. 1998;52:315–8. http://www.jstor.org/pss/2685433. Accessed 10 June 2010.

  2. Peixoto JL. Hierarchical variable selection in polynomial regression models. The American Statistician. 1987;41:311–3. http://www.jstor.org/pss/2684752. Accessed 10 June 2010.

  3. Peixoto JL. A property of well-formulated polynomial regression models. The American Statistician. 1990;44:26–30. http://www.jstor.org/pss/2684952. Accessed 10 June 2010.

I usually follow hierarchy but depart from it in some situations. For example, if you are testing tire wear versus mileage at several different speeds, your model might look like:

tread depth = intercept + mileage + mileage*speed

but it would not make physical sense to include a main effect of speed because the tire does not know what the speed will be at zero miles.

(On the other hand, you might still want to test for a speed effect because it might indicate that "break-in" effects differ at different speeds. On the other other hand, an even better way to handle break-in would be to get data at zero and at very low mileage and then test for non-linearity. Note that removing the intercept term can be thought of as a special case of violating hierarchy.)

I'll also reiterate what someone said above because it's very important: The authors need to make sure they know whether their software is centering the data. The tire model above becomes physically nonsensical if the software replaces mileage with (mileage - mean of mileage).

The same sorts of things are relevant in pharmaceutical stability studies (mentioned tangentially in "Stability Models for Sequential Storage", Emil M. Friedman and Sam C. Shum, AAPS PharmSciTech, Vol. 12, No. 1, March 2011, DOI: 10.1208/s12249-010-9558-x).

share|improve this answer
    
thank you, this is a great answer and will help me explain it to people who are not statistically savvy. –  djhocking May 9 '12 at 12:54
add comment

Lots of good answers already. There was a paper by Rindskopf on some cases where you do not need the main effects. (Also see this one)

share|improve this answer
add comment

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

I agree with Peter. I think the rule is folklore. Why could we conceive of a situation where two variables would affect the model only because of an interaction. An analogy in chemistry is that two chemicals are totally inert by themselves but cause an explosion when mixed together. Mathematical/statistical niceties like invariance have nothing to do with a real problem with real data. I just think that when there are a lot of variables to consider there is an awful lot of testing to do if you are going to look at all main effects and most if not all first order interactions. We also almost never look at second order interactions even in small experiments with only a handful of variables. The thinking is that the higher the order of interaction the less likely it is that there is a real effect. So don't look at first or second order interactions if the main effect isn't there. A good rule perhaps but to follow it religiously means overlooking the exceptions and your problem may be an exception.

share|improve this answer
5  
Re: "Mathematical/statistical niceties like invariance have nothing to do with a real problem with real data" - it does have to do with a real problem with real data when your $p$-values, and therefore your statistical inference (and therefore your "real world" decision about the importance of a predictor), can depend on something as arbitrary as the decision to center your predictors. –  Macro May 4 '12 at 19:59
    
I probably misspoke saying that invariance has no relevance in the real world. My intended point was that some mathematical results may not be relevant in a particular practical problem. As an example least squares estimates are maximum likelihood under normal error assumptions and by the Gauss Markov theorem are minimum variance unbiased under weaker conditions, but I wouldn't use it when there are outliers in the data. By the same token should a property like invariance rule out including an interaction when it makes sense say medically that it would occur without the main effects? –  Michael Chernick May 5 '12 at 18:09
add comment

I have had a real case that illustrates this. In the data one of the variables represented group with 0-control and 1-treatment. The other predictor represented time period with 0-before treatment and 1- after treatment. The interaction was the main parameter of interest measuring the effect of the treatment, the difference after the treatment in the treatment group above any effect of time measured in the control group. The main effect from group measured the difference in the 2 groups before any treatment, so it could easily be 0 (in a randomized experiment it should be 0, this on was not). The 2nd main effect measures the the difference between the before and after time periods in the control group where there was no treatment, so this also makes sense that it could be 0 while the interaction term is non-zero. Of course this depends on how things were coded and a different coding would change the meanings and if the interaction makes sense without the main effects. So it only makes sense to fit the interaction without the main effects in specific cases.

share|improve this answer
add comment

[trying to answer a part of the original question which seems left uncovered in most answers: "should AIC, as a model selection criterion be trusted?"]

AIC should be used more as a guideline, than a rule that should be taken as gospel.

The effectiveness of AIC (or BIC or any similar 'simple' criterion for model selection) highly depends on the learning algorithm, and the problem.

Think of it this way: the goal of the complexity (number of factors) term in the AIC formula is simple: to avoid selecting models which over-fit. But the simplicity of AIC very often fails to capture the real complexity of the problem itself. This is why there are other practical techniques to avoid over-fitting: for example, cross-validation or adding a regularization term.

When I use online SGD (stochastic gradient descent) to do linear regression on a data-set with a very large number of inputs, I find AIC to be a terrible predictor of model quality because it excessively penalizes complex models with large number of terms. There are many real life situations where each term has a tiny effect, but together, a large number of them gives strong statistical evidence of an outcome. AIC and BIC model-selection criteria would reject these models and prefer the simpler ones, even though the more complex ones are superior.

In the end, it is the generalization error (roughly: out of sample performance) that counts. AIC can give you some hint of model quality in some relatively simple situations. Just be careful and remember that real life is more often than not, more complex than a simple formula.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.