6
$\begingroup$

Considering that the interaction between two variables is significant, is it always the case that at least one of the two simple effects will be significant? If so, is there any proof or a counter example?

For the sake of simplicity, let's assume that we are talking here about 2 x 2 or 2 x continuous designs.


Edit

Example:

set.seed(1)
n <- 100
x    <- rep(c("a", "b"), each = n/2)
x.ab <- ifelse(x == "a", -.5, +.5)
x.a  <- ifelse(x == "a",  0, +1)
x.b  <- ifelse(x == "b",  0, -1)
z    <- rnorm(n)
y    <- x.ab*z + rnorm(n)

We now test the interaction:

> summary(lm(y ~ x.ab*z))

             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.037776   0.097254  -0.388    0.699    
x.ab         0.235133   0.194508   1.209    0.230    
z            0.001777   0.109290   0.016    0.987    
x.ab:z       0.946657   0.218581   4.331 3.65e-05 ***

The x.ab:z interaction is significant (and there are no main effects of neither x nor z), so we can test for simple effects:

> summary(lm(y ~ x.a*z))

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -0.1553     0.1375  -1.129  0.26151    
x.a           0.2351     0.1945   1.209  0.22969    
z            -0.4716     0.1659  -2.843  0.00546 ** 
x.a:z         0.9467     0.2186   4.331 3.65e-05 ***

The simple effect y ~ z is significant when x = a (p = 0.005).

> summary(lm(y ~ x.b*z))

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.07979    0.13754   0.580   0.5632    
x.b          0.23513    0.19451   1.209   0.2297    
z            0.47510    0.14235   3.338   0.0012 ** 
x.b:z        0.94666    0.21858   4.331 3.65e-05 ***

The simple effect y ~ z is also significant when x = b (p = 0.001).

As we can see in this example, both simple effects are significant. Is it always the case that at least one of the simple effects will be significant, given that the interaction is significant?

Ps: What I refer to as "main effects" and "simple effects" is based on Spiller et al. (2013).

$\endgroup$

1 Answer 1

8
$\begingroup$

Considering that the interaction between two variables is significant, is it always the case that at least one of the two simple effects will be significant?

No, not at all.

If so, is there any proof or a counter example?

Yes, it is easy to create a counter example:

> set.seed(1)
> N <- 200

> A <- rep(c(0,0,1,1), times = N/4)
> B <- rep(c(0,1), times = N/2)
> y <- A*B + rnorm(N)

> summary(lm(y ~ A*B))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept)  -0.0407     0.1317   -0.31   0.7579   
A             0.2250     0.1863    1.21   0.2287   
B             0.0789     0.1863    0.42   0.6723   
A:B           0.6971     0.2635    2.65   0.0088 **

I would always advise against relying on statistical significance, which relies on p-values (that depend on sample size) and arbitrary significance levels.


Edit:

Following the update, I have to say that I hope the procedure in the edited question is not something that is being used in practice, but anyway, yes of course we can find a counterexample. Just set the seed in your code to, for example, 276

$\endgroup$
10
  • $\begingroup$ In your example you are only testing one of the simple effect: For instance the Y ~ B simple effect when A = 0, which is not significant (p = .67). However, if you test for that same effect for A = 1, then Y ~ B is significant (p < .001). This is how: A.lvl1 <- ifelse(A == 1, 0, 1), then summary(lm(y ~ A.lvl1*B)). $\endgroup$
    – mat
    Jul 17, 2020 at 10:04
  • 6
    $\begingroup$ But that wasn't your question. You didnt mention a univariate regression of each of the main effects. My example shows that both main effects are not significant but the interaction is as per the question as written. If you want to clarify your question I will see if I can create a different example. $\endgroup$ Jul 17, 2020 at 10:11
  • $\begingroup$ I've updated my post to clarify what I meant. I think we're not using quite the same nomenclature. $\endgroup$
    – mat
    Jul 17, 2020 at 10:26
  • 2
    $\begingroup$ Well as I mentioned when I first answered, statistical procedures based around p values and arbitrary levels for them are not to be trusted. In my opinion they are close to worthless. The fact that we can retain the same code and just change the seed to get completely different "results" gives a strong hint. In this specific example I am not even sure what the purpose of the procedure is, so if you can explain that then perhaps I can be more specific about it. What research question is the procedure trying to answer ? $\endgroup$ Jul 17, 2020 at 12:33
  • 7
    $\begingroup$ I'm not saying we should be careful. I'm saying we shouldn't use them. A problem I see all the time is people agreeing that p values are bad, and then going on to apply procedures like stepwise regression and other abominations. One of the reasons I love lme4 is that it doesn't produce p values. I found it incredible how many people would complain about that. The day I discovered lmerTest was a very sad one. $\endgroup$ Jul 17, 2020 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.