6
$\begingroup$

Consider a linear regression (based on least squares) on two predictors including an interaction term: $$Y=(b_0+b_1X_1)+(b_2+b_3X_1)X_2$$

$b_2$ here corresponds to the conditional effect of $X_2$ when $X_1=0$. A common mistake is to understand $b_2$ as being the main effect of $X_2$, i.e. the average effect of $X_2$ over all possible values of $X_1$.

Now let's assume that $X_1$ was centered, that is $\overline{X_1}=0$. It becomes now true that $b_2$ is the average effect of $X_2$ over all possible values of $X_1$, in the sense that $\overline{b_2+b_3X_1}=b_2$. In such conditions, the meaning given to $b_2$ is nearly indistinguishable from the meaning that we would give to the effect of $X_2$ in a simple regression (where $X_2$ would be the only variable, let's call this effect $B_2$).

In practice, it seems that $b_2$ and $B_2$ are reasonably close to each other.

Question:

Are there any "common knowledge" examples of situations where $B_2$ and $b_2$ are remarkably far from each other?

Are there any known upper bounds to $|b_2-B_2|$?


Edit (came after @Robert Long's answer):

For the record, a very rough calculation of what the difference $|b_2-B_2|$ might look like.

$B_2$ can be computed via the usual covariance formula, giving $$B_2=b_2+b_3\dfrac{Cov(X_1X_2,X_2)}{Var(X_2)}$$ The last fraction is roughly distributed like the ratio of two normal variables, $\mathcal N(\mu,\frac{3+2\mu^2}{\sqrt N})$ and $\mathcal N(0,\frac{2}{\sqrt N})$ (not independent, unfortunately), assuming that $X_1\sim \mathcal N(0,1)$ and $X_2\sim \mathcal N(\mu,1)$. I've asked a separate question to try to circumvent my limited calculation skills.

$\endgroup$
3
  • $\begingroup$ (+1) Interesting question Intuitively I think your statement that b_2 and B2 are close isn't correct but I am looking into it :) $\endgroup$ Commented Jul 21, 2020 at 5:50
  • $\begingroup$ @RobertLong Oh I've just noticed your edit. I'd always thought that "factor" was a synonym for "explanatory variable" - just like "predictor". $\endgroup$ Commented Jul 21, 2020 at 13:33
  • 1
    $\begingroup$ I guess it can be used that way but by far the most common use of "factor" is as another word for a "categorical variable." $\endgroup$ Commented Jul 21, 2020 at 14:09

2 Answers 2

5
$\begingroup$

$b_2$ here corresponds to the conditional effect of $X_2$ when $X_1=0$. A common mistake is to understand $b_2$ as being the main effect of $X_2$, i.e. the average effect of $X_2$ over all possible values of $X_1$.

Indeed. I typically answer at least one question per week where this mistake is made. It it also worth pointing out for completeness that $b_1$ here corresponds to the conditional effect of $X_1$ when $X_2= 0 $ and not the main effect of $X_1$ which is easily seen by rearranging the formula

$$Y=(b_0+b_2X_2)+(b_1+b_3X_2)X_1$$

In practice, it seems that $b_2$ and $B_2$ are reasonably close to each other.

I think this is false in general for this model and will will only be true when the interaction term $b_3$ is very small.

Are there any "common knowledge" examples of situations where $B_2$ and $b_2$ are remarkably far from each other?

Yes, when the $b_3$ is meaningfully large then $B_2$ and $b_2$ will be meaningfully apart. I am thinking of how to show this algebraiclly and graphically but I don't have much time now, so I will resort to a simple simulation for now. First with no interaction:

> set.seed(25)
> N <- 100
> 
> dt <- data.frame(X1 = rnorm(N, 0, 1), X2 = rnorm(N, 5, 1))
> 
> X <- model.matrix(~ X1 + X2 + X1:X2, dt)
> 
> betas <- c(10, -2, 2, 0)
> 
> dt$Y <- X %*% betas + rnorm(N, 0, 1)
> 
> (m1 <- lm(Y ~ X1*X2, data = dt))$coefficients[3]
  X2 
2.06 
> (m2 <- lm(Y ~ X2, data = dt))$coefficients[2]
  X2 
1.96

as expected. And now with an interaction:

> set.seed(25)
> N <- 100
> 
> dt <- data.frame(X1 = rnorm(N, 0, 1), X2 = rnorm(N, 5, 1))
> 
> X <- model.matrix(~ X1 + X2 + X1:X2, dt)
> 
> betas <- c(10, -2, 2, 10)
> 
> dt$Y <- X %*% betas + rnorm(N, 0, 1)
> 
> (m1 <- lm(Y ~ X1*X2, data = dt))$coefficients[3]
  X2 
2.06 
> (m2 <- lm(Y ~ X2, data = dt))$coefficients[2]
  X2 
3.29 

Are there any known upper bounds to $|b_2-B_2|$

I don't think so. As you increase $|b_3|$ then $|b_2-B_2|$ should increase

$\endgroup$
8
  • $\begingroup$ I see: when $b_2$ is small with respect to the interaction effect, meaning the average slope is close to $0$ compared to the extremal slopes, then $B_2$ is likely to be (statistically insignificant and) very sensitive to error. It makes perfect sense, thank you very much. As for the upper bound I would still be happy with something that depends on $b_3$. But in light of your answer, this second question matters less. I think it would also be interesting to have some kind of estimates of how sensitive $B_2$ becomes in such situations. $\endgroup$ Commented Jul 21, 2020 at 8:23
  • $\begingroup$ I agree. I need to expand this answer. The standard error for $B_2$ will become large as the interaction gets larger. In the continuous x continous case here the confidence interval gets very wide (actually containing $b_2$ in the simulations I looked at). I will look at this some more later today as it's very interesting, and also look at categorical $X1$ and $X2$ $\endgroup$ Commented Jul 21, 2020 at 8:30
  • $\begingroup$ Regarding the possible values for $|𝑏_2βˆ’π΅_2|$, in fact using the equation we can get $$𝐡_2=𝑏_2+𝑏_3\dfrac{πΆπ‘œπ‘£(𝑋_1𝑋_2,𝑋_2)}{π‘‰π‘Žπ‘Ÿ(𝑋_2)}$$ which does support the fact that the absolute difference grows with $|𝑏_3|$. I believe that the law of the sample covariance of $𝑋_1𝑋_2$ and $𝑋_2$ can be derived from the CLT, which settles this part of the question. $\endgroup$ Commented Jul 21, 2020 at 11:53
  • $\begingroup$ Ahh that's very useful. I will incorporate it into my simulations later. $\endgroup$ Commented Jul 21, 2020 at 11:59
  • $\begingroup$ In an orthogonal factorial design (i.e., a balanced design with equal sample sizes in each combination of X1 and X2), B2 will equal b2. In this case, the covariances are 0. Thus, using @ArnaudMortier's equation, b3 drops out. In an unbalanced design, centering the variables reduces but does not eliminate, the covariances, thus resulting in a B2 that is generally closer to b2. $\endgroup$
    – dbwilson
    Commented Jul 21, 2020 at 12:08
2
$\begingroup$

Adding to @RobertLong's answer, there is a slight conceptual mistake in the way $b_2$ is described in the question in the case where $X_1$ was centered. It is indeed true that $b_2$ becomes the average effect of $X_2$ over all possible values of $X_1$, in the sense that $\overline{b_2+b_3X_1}=b_2$, but it should be emphasized that this is an average of simple effects. It may have nothing to do with the main effect of $X_2$ on the DV, which means that $b_2$ may be really far from $B_2$ even without interaction.

Here is an example where there is no interaction, and $b_2$ and $B_2$ have nothing in common: the vertical axis is the DV $Y$, the horizontal axis is for the covariate $X_2$, and the colors stand for levels of the covariate $X_1$. For any value of $X_1$, the simple effect $b_2+b_3X_1$ is around $-1$, while the main effect $B_2$ is clearly positive.

enter image description here

$\endgroup$
1
  • $\begingroup$ (+1) Interesting !! I will try t take a look at this in more detail after you add the simulation code. $\endgroup$ Commented May 6, 2021 at 14:17

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.