# Why main effect becomes non-significant after reversing moderator?

I am currently fitting a linear regression with two IVs and their interaction effect:

Y = β1X1 + β2X2 + β3X1*X2

The data was collected in an experimental study. X1 is a dummy variable, taking the values 1 or 2 depending on which of two experimental groups study participants were randomly assigned to. X2 was measured on a 1 to 7 point Likert-scale.*

When I run this regression, X1 is significant (p < 0.05). However, when I reverse X2, i.e. X2b = 8 - X2, X1 is no longer significant.

Why could reversing a Likert scale have such an impact?

R output before reversing X2:

 > summary(lm(Y ~ X1*X2, data = data))

Call:
lm(formula = Y ~ X1*X2, data = data)

Residuals:
Min      1Q  Median      3Q     Max
-3.9407 -0.8596  0.2539  1.2214  3.2453

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)      2.38767    1.09785   2.175   0.0301 *
X1               1.31707    0.59331   2.220   0.0269 *
X2               0.10735    0.23936   0.448   0.6540
X1:X2           -0.09422    0.12921  -0.729   0.4662


R output after reversing X2:

> summary(lm(Y ~ X1*X2b, data = data))

Call:
lm(formula = interface_playful ~ c_0001_pool * finwb, data = bfi3)

Residuals:
Min      1Q  Median      3Q     Max
-3.9407 -0.8596  0.2539  1.2214  3.2453

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)        3.24651    0.93832   3.460 0.000588 ***
X1                 0.56332    0.50558   1.114 0.265737
X2b               -0.10735    0.23936  -0.448 0.653994
X1:X2b             0.09422    0.12921   0.729 0.466240
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


*This info was updated on September 14, 2021. Previously, we inadvertently stated that both IVs were measured on a 1-7 Likert scale.

This is an example of dummy regression. Because X1 is the dummy variable (has values of 0 or 1), the coefficient of X1 is equal to to the difference in the intercept (predicted value of y for X2 = 0) of the group with X1 = 1 compared to the group with X1 = 0. The p-value corresponds to the null hypothesis whether this difference in intercepts is equal to 0. If you reverse the X2 variable, this difference in intercepts will change (because the slopes in both groups change their sign). In your example the X1 coefficient is far away from 0 in the original model and thus significant. In the reversed model, the coefficient is closer to 0 and thus not significant. I simulated data quite similar to yours below. Note that I used a big sample size, so all my coefficients are significant for both models.

> library(sjPlot)
> library(ggplot2)
> # simulate data
> set.seed(1)
> N <- 10000
> X1 <- sample(0:1, size = N, replace = TRUE)
> X2 <- sample(1:7, size = N, replace = TRUE)
> X2_rev <- 8 - X2
> Y <- 2.4 + 1.3 * X1 + 0.1 * X2 + -0.1 * X1 * X2 + rnorm(N)
> d <- data.frame(X1, X2, X2_rev, Y)
> # fit original model
> mod1 <- lm(Y ~ X1*X2, data = d)
> summary(mod1)

Call:
lm(formula = Y ~ X1 * X2, data = d)

Residuals:
Min      1Q  Median      3Q     Max
-4.3027 -0.6817 -0.0101  0.6657  3.7270

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  2.404209   0.031651  75.959   <2e-16 ***
X1           1.287144   0.044422  28.975   <2e-16 ***
X2           0.098386   0.007058  13.940   <2e-16 ***
X1:X2       -0.095524   0.009901  -9.648   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.996 on 9996 degrees of freedom
Multiple R-squared:  0.1839,    Adjusted R-squared:  0.1837
F-statistic: 750.8 on 3 and 9996 DF,  p-value: < 2.2e-16

> plot_model(mod1, type = "pred", terms = c("X2", "X1"), axis.lim = list(c(0,7), c(2,4)))


We see in the plot that the intercept for group X1 = 0 is about 2.4 while the intercept for group X1 = 1 is about 2.4 + 1.3 = 3.7 which is in line with the coefficient estimates from the table above.

> # fit model with reversed X2 variable
> mod2 <- lm(Y ~ X1*X2_rev, data = d)
> summary(mod2)

Call:
lm(formula = Y ~ X1 * X2_rev, data = d)

Residuals:
Min      1Q  Median      3Q     Max
-4.3027 -0.6817 -0.0101  0.6657  3.7270

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  3.191293   0.031469 101.411   <2e-16 ***
X1           0.522951   0.044240  11.821   <2e-16 ***
X2_rev      -0.098386   0.007058 -13.940   <2e-16 ***
X1:X2_rev    0.095524   0.009901   9.648   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.996 on 9996 degrees of freedom
Multiple R-squared:  0.1839,    Adjusted R-squared:  0.1837
F-statistic: 750.8 on 3 and 9996 DF,  p-value: < 2.2e-16

> plot_model(mod2, type = "pred", terms = c("X2_rev", "X1"), axis.lim = list(c(0,7), c(2,4)))


Now the intercept for group X1 = 0 is about 3.2 and the intercept for group X1 = 1 is about 3.2 + 0.5 = 3.7

As expected, the slope in both groups flipped sign (we cannot realy see this for group X1 = 1 because the slope is almost 0).

There are no main effects in a model with an interaction. You cannot conclude something about the effect of X2, or X2b without also looking at the interaction. In both cases, the standard error for the interaction is identical, as is the estimate, albeit with a flipped sign.

Your data provides very little evidence for interaction, though, so the coefficients of a main effects model (without interaction) may be more interesting. If you do use a main effects model after this, you should correct the $$p$$-values for having looked at both models though.

The ordinary least squares regression you are running tries to fit a regression line that represents your measured values the best i.e. it wants to minimise the sum of squared differences between your observed data and the predicted on the line. Thus, although you inverse the values of your factor b so that the absolute value of the estimate (slope) stays the same, the regression line needs to adjust (with the inverse values for b) also for the values of variable a so that squared residuals remain minimised. This leads to a different estimate (slope) for variable a and a different p-value. You would need to inverse also variable a in order to obtain the same (absolute) estimate values.