9
$\begingroup$

Should two continuous variables be correlated to include their interaction? I.e. does y = x1 + x2 + x1x2 make sense if correlation between x1 and x2 is weak at -0.1, and their coefficients in the regression have opposite signs (x1 β1 close to +1 and x2 β2 close to -1)?

And if the estimated coefficient on x1x2 is then +3, does this effectively mean the total effect of x2 is actually positive (with β3 = +3 offsetting β2 = -1), thus both variables have a net positive association with y?

Lastly, if trying to represent this with scatter plots like pairs(y x1 x2 x1x2) in r, wouldn't individual and time fixed effects (panel data) make any relationship impossible to see visually?

Thanks!

$\endgroup$
3
  • $\begingroup$ "correlated" should be removed from the title since independent implies uncorrelated $\endgroup$
    – TrungDung
    Feb 17 at 13:19
  • 1
    $\begingroup$ @TrungDung I disagree. While lack of correlation is implied by independent, one could be interested in variables that are uncorrelated but dependent. It is up to Junlei to decide on the question to ask. $\endgroup$
    – Dave
    Feb 17 at 13:23
  • $\begingroup$ I think a linguistic issue is that in regression of a model $Y=\beta_0+\beta_1X_1+\beta_2X_2+\epsilon$, you have $X_1$ and $X_2$ called independent variables even if they are not independent of each other (and clearly not of $Y$) $\endgroup$
    – Henry
    Feb 20 at 16:52

2 Answers 2

10
$\begingroup$

Definitely!

set.seed(2022)
N <- 10
x1 <- rep(c(0, 1), N) # (0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1)
x2 <- c(rep(0, N), rep(1, N)) # (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
x12 <- x1 * x2 # interaction between x1 and x2
#
y <- x1 - x2 + 5*x12 + rnorm(length(x1))

In this example, $x_1$ and $x_2$ are independent (not just uncorrelated) and have opposite signs in the regression model. However, the interaction term is important. Compare the $R^2$ or adjusted $R^2$ your models have when you regress on just $x_1$ and $x_2$ versus regressing on those variables plus their interaction.

$\endgroup$
2
  • 1
    $\begingroup$ What do you mean by "independent (not just uncorrelated)" here? I know the term "independent" only for random variables, but not for data. $\endgroup$
    – jochen
    Feb 18 at 14:58
  • $\begingroup$ @jochen By looking at either x1 or x2, you get absolutely no insight into the other. $\endgroup$
    – Dave
    Feb 22 at 20:00
5
$\begingroup$

To slightly add to Dave's +1 answer, recall that omitting a regressor from a regression model will not affect the remaining coefficients (something that I would argue is more relevant and less trivial than a "better" (higher) $R^2$, which will always obtain with additional regressors - although Dave of course also points to adjusted $R^2$) if it's uncorrelated (orthogonal) with the remaining regressors.

To see this, write the OLS estimate containing two blocks of regressors $X_1$ and $X_2$ (in our example, $X_1$ could consist of x^2 and $X_2$ of x1 and x2) as $$ \hat\beta=\begin{pmatrix}X_1'X_1&X_1'X_2\\ X_2'X_1&X_2'X_2\end{pmatrix}^{-1}\begin{pmatrix}X_1'y\\ X_2'y\end{pmatrix} $$ That is, if the two blocks are orthogonal ($X_1'X_2=0$), we obtain \begin{eqnarray*} \hat\beta&=&\begin{pmatrix}X_1'X_1&0\\ 0&X_2'X_2\end{pmatrix}^{-1}\begin{pmatrix}X_1'y\\ X_2'y\end{pmatrix}\\ &=&\begin{pmatrix}(X_1'X_1)^{-1}&0\\ 0&(X_2'X_2)\end{pmatrix}\begin{pmatrix}X_1'y\\ X_2'y\end{pmatrix}\\ &=&\begin{pmatrix}(X_1'X_1)^{-1}X_1'y\\ (X_2'X_2)^{-1}X_2'y\end{pmatrix}, \end{eqnarray*} i.e. the coefficients of two separate regressions.

Now, in Dave's example, the interaction has its own partial effect (equal to 5), and, while x1 and x2 are uncorrelated, x12 is not uncorrelated with x1 nor x2. I slightly extend Dave's example:

set.seed(2022)
N <- 10
x1 <- rep(c(0, 1), N) # (0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1)
x2 <- c(rep(0, N), rep(1, N)) # (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
x12 <- x1 * x2 # interaction between x1 and x2
#
y <- x1 - x2 + 5*x12 + rnorm(length(x1))

lm.int <- lm(y ~ x1 + x2 + x12)
lm.wo.int <- lm(y ~ x1 + x2)
summary(lm.int)
summary(lm.wo.int)

cor(x1, x2)  # zero
cor(x12, x2) # not zero
cor(x12, x1) # not zero
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.