Interactions make terms significant in regression when they should not be

Question

I am writing code to prepare for running a logistic regression on real data. I have sample data for all my IVs but not for the outcome variable. There are many strong dependencies among the IVs but I have a lot of data points.

I created fake outcome data that is dependent only on a single IV main effect. The regression without interactions came out as I expected, only the single IV was significant, and the p-value was extremely low. However, when I update the formula to include all two-way interactions, the result is crazy. A large number of main effects and interactions are significant, some with fairly low p-values.

Why is this happening? And is there anything I can do about it?

Would appreciate any insight you have! Thanks!

---

Update 2: I have found that performing backward stepwise regression using BIC gets me down to the only factor that I made significant, so I hope that if I use that method with real data it will work out. I am still looking for some insight into what is happening here.

model2 <- step(model2.start,
               direction = "backward",
               k = log(nrow(model1.data)))
print(summary(model2))

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) -0.83138    0.02277 -36.515   <2e-16 ***
genderM      0.29235    0.03511   8.326   <2e-16 ***

---

Update: Added code and results for Peter Flom.

Formula with no interactions:

model1.start <- glm(formula = model1.formula,
                    data = model1.data,
                    family = binomial(link=logit))
print(summary(model1.start))

Coefficients:
                   Estimate Std. Error z value Pr(>|z|)    
(Intercept)      -0.9597428  0.1336164  -7.183 6.83e-13 ***
userChangeCount   0.0069607  0.0257332   0.270    0.787    
genderM           0.2975069  0.0356766   8.339  < 2e-16 ***
birth_year       -0.0009198  0.0019762  -0.465    0.642    
statusG           0.0559423  0.0748278   0.748    0.455    
statusN          -0.0646233  0.1669781  -0.387    0.699    
statusS           0.0187706  0.0662441   0.283    0.777    
statusU          -0.0257740  0.0832096  -0.310    0.757    
collegeA          0.0129889  0.0679490   0.191    0.848    
collegeB         -0.0040121  0.0788700  -0.051    0.959    
collegeC         -0.1461340  0.0899802  -1.624    0.104    
collegeD          0.0331471  0.0863881   0.384    0.701    
collegeE          0.0453438  0.0756112   0.600    0.549    
collegeF          0.0848041  0.0697141   1.216    0.224    
collegeG          0.0901069  0.0849070   1.061    0.289    

Using update.formula to add in interactions:

model2.formula <- update.formula(model1.formula, ~ .^2)
model2.start <- glm(formula = model2.formula,
                   data = model1.data,
                   family = binomial(link=logit))
print(summary(model2.start))

Coefficients: (8 not defined because of singularities)
                                   Estimate Std. Error z value Pr(>|z|)    
(Intercept)                      -2.2344367  0.5212149  -4.287 1.81e-05 ***
userChangeCount                   0.0212035  0.2069908   0.102 0.918410    
genderM                           0.7479265  0.2957677   2.529 0.011447 *  
birth_year                       -0.0412143  0.0168060  -2.452 0.014192 *  
statusG                           2.6235509  0.8689383   3.019 0.002534 ** 
statusN                           0.9568523  0.5929653   1.614 0.106598    
statusS                           0.2330824  0.5272580   0.442 0.658442    
statusU                           1.1998468  0.6039849   1.987 0.046972 *  
collegeA                         -1.4456800  0.8844914  -1.634 0.102159    
collegeB                          0.3037282  0.3369018   0.902 0.367305    
collegeC                         -0.5860054  0.7721724  -0.759 0.447909    
collegeD                         -0.4193368  0.3765112  -1.114 0.265389    
collegeE                         -0.0973697  0.3518518  -0.277 0.781984    
collegeF                          0.5329413  0.2795513   1.906 0.056596 .  
collegeG                          1.2383888  0.3673590   3.371 0.000749 ***
userChangeCount:genderM          -0.0743210  0.0550317  -1.351 0.176852    
userChangeCount:birth_year       -0.0009157  0.0031425  -0.291 0.770763    
userChangeCount:statusG           0.2469431  0.1298367   1.902 0.057177 .  
userChangeCount:statusN          -0.5741316  0.3610974  -1.590 0.111843    
userChangeCount:statusS           0.1604922  0.1174547   1.366 0.171808    
userChangeCount:statusU           0.2652325  0.1428424   1.857 0.063337 .  
userChangeCount:collegeA          0.0428316  0.1007660   0.425 0.670793    
userChangeCount:collegeB          0.0720982  0.1183963   0.609 0.542553    
userChangeCount:collegeC          0.1104534  0.1214471   0.909 0.363098    
userChangeCount:collegeD         -0.1389037  0.1487849  -0.934 0.350517    
userChangeCount:collegeE          0.1559318  0.1078295   1.446 0.148150    
userChangeCount:collegeF          0.0575791  0.1017219   0.566 0.571364    
userChangeCount:collegeG          0.0324145  0.1400930   0.231 0.817021    
genderM:birth_year                0.0080877  0.0042729   1.893 0.058388 .  
genderM:statusG                  -0.1330119  0.1663472  -0.800 0.423940    
genderM:statusN                  -0.1387854  0.3548589  -0.391 0.695723    
genderM:statusS                   0.1615500  0.1472553   1.097 0.272609    
genderM:statusU                  -0.1702773  0.1870952  -0.910 0.362764    
genderM:collegeA                  0.0496312  0.1399796   0.355 0.722919    
genderM:collegeB                 -0.0238660  0.1620387  -0.147 0.882907    
genderM:collegeC                  0.2330523  0.1905078   1.223 0.221208    
genderM:collegeD                 -0.0220152  0.1874495  -0.117 0.906507    
genderM:collegeE                 -0.2661413  0.1569381  -1.696 0.089917 .  
genderM:collegeF                 -0.0251615  0.1491900  -0.169 0.866068    
genderM:collegeG                  0.1045658  0.1781036   0.587 0.557132    
birth_year:statusG                0.0007200  0.0080201   0.090 0.928468    
birth_year:statusN                0.0723133  0.0308732   2.342 0.019167 *  
birth_year:statusS                0.0046791  0.0055686   0.840 0.400759    
birth_year:statusU               -0.0151671  0.0187630  -0.808 0.418889    
birth_year:collegeA               0.0068271  0.0087997   0.776 0.437845    
birth_year:collegeB               0.0203172  0.0095400   2.130 0.033197 *  
birth_year:collegeC               0.0168092  0.0105057   1.600 0.109596    
birth_year:collegeD              -0.0093096  0.0106617  -0.873 0.382563    
birth_year:collegeE              -0.0060650  0.0076375  -0.794 0.427128    
birth_year:collegeF               0.0144749  0.0073667   1.965 0.049424 *  
birth_year:collegeG               0.0409526  0.0109024   3.756 0.000172 ***
statusG:collegeA                 -0.0161367  0.3181531  -0.051 0.959549    
statusN:collegeA                         NA         NA      NA       NA    
statusS:collegeA                  0.4597889  0.2774546   1.657 0.097486 .  
statusU:collegeA                 -0.2270056  0.3392752  -0.669 0.503438    
statusG:collegeB                 -0.4238954  0.3538770  -1.198 0.230971    
statusN:collegeB                         NA         NA      NA       NA    
statusS:collegeB                  0.4453308  0.3020272   1.474 0.140354    
statusU:collegeB                 -0.4386064  0.3661169  -1.198 0.230919    
statusG:collegeC                 -1.0156894  0.3707719  -2.739 0.006155 ** 
statusN:collegeC                         NA         NA      NA       NA    
statusS:collegeC                 -0.7283356  0.3466765  -2.101 0.035649 *  
statusU:collegeC                         NA         NA      NA       NA    
statusG:collegeD                  0.4933988  0.3977650   1.240 0.214817    
statusN:collegeD                         NA         NA      NA       NA    
statusS:collegeD                  0.1421565  0.3152644   0.451 0.652053    
statusU:collegeD                  0.5731627  0.4181356   1.371 0.170450    
statusG:collegeE                  0.2283331  0.3918929   0.583 0.560135    
statusN:collegeE                         NA         NA      NA       NA    
statusS:collegeE                  0.3603496  0.3223417   1.118 0.263605    
statusU:collegeE                  0.3225674  0.4006952   0.805 0.420808    
statusG:collegeF                 -0.3994595  0.2919074  -1.368 0.171172    
statusN:collegeF                         NA         NA      NA       NA    
statusS:collegeF                  0.1203146  0.2477903   0.486 0.627286    
statusU:collegeF                 -0.7253529  0.3232596  -2.244 0.024841 *  
statusG:collegeG                 -1.3172171  0.3663165  -3.596 0.000323 ***
statusN:collegeG                         NA         NA      NA       NA    
statusS:collegeG                 -0.1037409  0.3286267  -0.316 0.752245    
statusU:collegeG                 -1.3636992  0.4203883  -3.244 0.001179 ** 

As you can see, now there are many significant effects, including main effects. To generate the fake data, I generated the outcome randomly and then went back and set a random 10% of males to have a positive outcome.

Answer 1

The coefficient for the lower order term is showing the unique contribution of that variable the prediction of the dependent variable, controlling for all of the other variables in the model. Whether you are entering interaction terms or other lower order main effects, the coefficient for the lower order term will usually change to reflect the fact that other predictors are in the model. An added issue in the model that you present is that you have a large number of predictors and interactions, some of which may attain significance by chance. A common practice in testing interaction terms involves centering the lower order variables around their mean and then computing the cross products based on the centered variables. Centering the variables is intended to reduce the linear association between the lower order terms and their cross products, but the problems you identify are likely to persist in the overall model. Ultimately, you need to test a simpler model in which the main effects and interactions are chosen more selectively. Good resources to read concerning the impact of interaction terms on main effects in regression include:

Kromrey, J.D., Foster-Johnson, L. (1998). Mean Centering in Moderated Multiple Regression: Much Ado About Nothing. Educational and Psychological Measurement, 58(1), 42-67.

Echambadi, R. (2007). Mean centering does not alleviate collinearity problems in moderated multiple regression models. Marketing Science.

Edwards, J.R. (2009). Seven deadly myths of testing moderation in organizational research. In Statistical and methodological myths and Urban Legends: Doctrine, Verity and Fable in the organizational and social sciences.

Shieh, G. (2011). Clarifying the role of mean centering in multicollinearity of interaction effects. British journal of mathematical and statistical psychology. 64(3), 462-477.

Answer 2

Including an interaction term changes the interpretations of the lower-order coefficients. Specifically, your lower order coefficients are now interpreted as the effect of that variable at 0 on all the other variables included in the interaction term. So, if you have allowed IV1 to interact with IV2, the "main effect" of IV1 should be interpreted as the effect of IV1 at 0 on IV2. Because 0 will not necessarily be an interpretable value, this can create a spuriously "significant" main effect that does not make substantive sense.

The typical way of dealing with this issue is to mean-center all the IVs that are allowed to interact. In other words, if you're fitting the following model:

mod <- lm(dv ~ iv1 * iv2 * iv3, data = d)

Do the following before fitting the model:

d$iv1 <- d$iv1 - mean(d$iv1)
d$iv2 <- d$iv2 - mean(d$iv2)
d$iv3 <- d$iv3 - mean(d$iv3)

The simple effects of iv1, iv2, and iv3 in mod will now be the main effects of iv1, iv2, and iv3.