Logistic regression interaction only significant when main effects are removed

Question

I'm interested in understanding if the interaction of a categorical variable (group A or B) and a continuous one (X1) can predict reaction time.

I started by testing the main effects plus the interaction using fitglm in Matlab.

modelspec = 'RT ~ group*X1';          % test interaction and individual factors

% Logistic regression model
mdl1 = fitglm(data,modelspec,'Distribution','binomial','CategoricalVars',[1])

With this model the interaction term is not significant, although when I plot the data it seem like there is an interaction. Model output:

Estimated Coefficients:
                   Estimate       SE        tStat      pValue  
                   ________    ________    _______    _________
    (Intercept)    -0.80295     0.15055    -5.3336    9.631e-08
    group           0.31        0.23212     1.3355    0.18171
    X1              0.1202      0.056139    2.1412    0.032261
    group:X1        0.13312     0.085264    1.5613    0.11846

1369 observations, 1365 error degrees of freedom
Dispersion: 1
Chi^2-statistic vs. constant model: 58.7, p-value = 1.12e-12

Next I looked at the interaction only and the interaction is significant:

modelspec = 'RT ~ group:X1';          % test interaction only

% Logistic regression model
mdl2 = fitglm(data,modelspec,'Distribution','binomial','CategoricalVars',[1])


Estimated Coefficients:
                   Estimate       SE       tStat       pValue  
                   ________    ________    ______    __________
    (Intercept)    -0.52461    0.071806    -7.306    2.7524e-13
    group:X1        0.26365    0.036835    7.1576    8.2125e-13

1369 observations, 1367 error degrees of freedom
Dispersion: 1
Chi^2-statistic vs. constant model: 54.1, p-value = 1.94e-13

I'm a bit lost and don't really know how to interpret these results, particularly from the first model (mdl1 = main effects and interactions). As I said, my main interest is the interaction. Should I report the model with the interaction only? What would be the correct way forward?

Any input is welcome!

Answer 1

Interactions in nonlinear models can get very tricky since these models can allow much more heterogeneity/flexibility in response. I always try to leave my linear intuition behind at home when I go out into the wild, nonlinear world, and do some math. So here we go.

In a logit with two main effects and an interaction,

$$Pr[y = 1 \vert x,z] = \frac{\exp (\alpha + \beta x + \gamma z + \delta z \cdot x)}{1+\exp (\alpha + \beta x + \gamma z + \delta z \cdot x)}=p. $$

After some tedious calculus and simplification, the partial of that with respect to $x$ becomes

$$ \frac{\partial Pr[y=1 \vert x,z]}{\partial x} = (\beta + \delta \cdot z) \cdot p \cdot (1-p).$$

This tells you the change in probability associated with a small increase in $x$. The term $p \cdot (1-p)$ is just a product of two probabilities, so it's a random number in $[0,0.25]$, so we can focus on the first term.

Note that the whole marginal effect (ME) depends on $z$ (as well as $x$), and that even the sign could be ambiguous, and that this can vary from observation to observation depending on covariates. Now there are two ways to proceed from here.

To determine if $z$ modifies the effect of $x$ you can calculate the MEs for a range of plausible values of $z$, use the delta-method to get variance-covariance matrix for those MEs, and then construct a hypothesis test that the MEs are all equal. If you reject that null, you can say that the data is inconsistent with "no interaction".

You could also try to differentiate the ME with respect to $z$ and use delta-method to calculate the SE of that. The cross-partial derivative is

$$ \frac{\partial Pr[y=1 \vert x,z]}{\partial x \partial z} = \delta \cdot p \cdot (1-p) +(\beta + \delta \cdot z) \cdot \left[ p \cdot (1-p)^2 - p^2 \cdot (1-p) \right].$$

This is even harder to untangle, but corresponds to a simpler counterfactual of bumping everyone's observed $z$ by an epsilon rather than resetting to a entirely different value. This hypothesis would be easier to test since it's just one number and you don't have to pick counterfactual values of $z$ (how many, which values, etc).

Your $x$ is binary, so it might make sense to replace differentiation with finite differences in the first step.

Also, all these derivatives can vary across observations, so it often makes sense to average them and do the hypothesis test on those AMEs.

All this is just a mathy, long-winded way to say that it is not sufficient to consider the statistical significance of the coefficient on the interaction.

For contrast, the linear case is much more straightforward. The ME of $x$ is $\beta + \delta \cdot z$, and the cross-partial of that is easy to calculate. It's just $\delta$, and it tells you everything you need to know about how the effect of $x$ depends on $z$ (sign, economic and statistical significance). But there's no heterogeneity whatsoever, which is the price you pay for the simplicity.

I suppose you can also try to interpret the index function coefficients as effects on the log odds (rather than on the probabilities), since that is still linear, but I find those harder to explain, both to myself and my colleagues, so I abstain from that.

Answer 2

Most of the time you should not include the interaction without the main effects. Check out here why. I also found this comment from @Thomas Levine under the same question that might help you:

If the interactions are only significant when the main effects are not in the model, it may be that the main effects are significant and the interactions not. Consider one highly significant main effect with variance on the order of 100 and another insignificant main effect for which all values are approximately one with very low variance. Their interaction is not significant, but the interaction effect will appear to be significant if the main effects are removed from the model