13
$\begingroup$

Suppose you fit a model $y = x_1 + x_2 + x_1\times x_2$. Are there any practical implications for estimation of the interaction effect if $x_1$ and $x_2$ are correlated?

I understand there could be collinearity issues if $x_1$ and $x_2$ are very correlated but that shouldn't affect the interaction term right?

$\endgroup$
3
  • 1
    $\begingroup$ You seem to be fishing for information about the correlation between $x_1x_2$ and $x_1$ when $x_1$ and $x_2$ are correlated. One way to get a sense of what can be deduced is to notice that although adding a constant (say $c$) to either of the $x_i$ will not change their correlation, it will change $x_1x_2$ into a constant plus $(x_1 x_2 + cx_1 + cx_2).$ Those last two terms show that $c$ has a profound effect on the correlation between $x_1x_2$ and $x_i.$ If this doesn't immediately suggest an answer to whatever your question might be, consider drawing some scatterplots. $\endgroup$
    – whuber
    Dec 27, 2018 at 14:04
  • $\begingroup$ @whuber I'm having trouble following your logic- is there a more explicit step by step explanation you can link to? I tried writing it out with the correlation formula, but was unable to reproduce your answer $\endgroup$
    – Hank Lin
    Dec 27, 2018 at 14:21
  • $\begingroup$ @whuber Also, regarding my original question i think some context might help as I agree it is quite vague. What happened was I presented my results looking for an interaction effect to a statistician I worked with and the first thing he asked me was whether the two predictors in my interaction were correlated. I had not examined the correlation and I asked him why it mattered. He could not quite explain why but said it did matter, hence my question. $\endgroup$
    – Hank Lin
    Dec 27, 2018 at 14:24

1 Answer 1

17
$\begingroup$

There is a reason your statistical consultant could not explain why introducing an interaction into a linear model might adversely affect the correlation structure: it depends on the circumstances and it is not generally true that there is an adverse effect. Just look at the datasets shown in the scatterplot matrices below to see all the different ways two variables might be related to their products.

The rest of this post explains how those figures were produced and might provide more insight into the situation.


First, let's get the obvious out of the way: writing $x_3=x_1x_2,$ you have a multiple regression involving the three variables $x_1, x_2, x_3.$ Whether or not there are collinearity problems depends on the linear relationships among the $x_i.$ That is universal.

What is special about this problem is the relationship between $x_3$ and the other $x_i;$ namely, that $x_3 = x_1x_2.$ Thus, if anyone has advised you to be careful, it must be due to an expectation that this multiplicative relationship mathematically entails some kind of multicollinearity among all the $x_i.$

This just isn't so, as can be demonstrated by exhibiting all possible patterns. I don't want to exhaust you with the pedantry of going through all the possibilities, so let me just sketch some of the most illustrative ones. The basic tool I will wield in this study is the observation that the correlation between any variables $x_1, x_2$ remains unchanged when the $x_i$ separately undergo linear transformations. That is, we may freely multiply either variable by constants and add other constants to the results without changing the correlation. However, these operations can profoundly alter the correlations between $x_1x_2$ and $x_i.$

(Nearly) constant product

It is possible for $x_1x_2$ to be constant (which, when a regression includes a constant, will be problematic). To create an example, simply generate nonzero values for $x_1$ and define $x_2 = c/x_1.$ Their product equals $c$ by construction.

You can perturb this example by changing $c\ne 0$ into a random variable with values close to $c.$ Doing this will introduce a small bit of correlation between the $x_i$ and their product, but not much. Here, for instance, is an example where $x_1$ is drawn from a Gamma$(5)$ distribution and $c$ has a Normal distribution with mean $1$ and standard deviation of just $1/100:$

Figure 0

Although the $x_i$ have a correlation of $\rho_{1\cdot 2}=-0.87$ in this example, their correlations with $x_1x_2$ are only $-0.06$ and $0.00.$

Hence, although there may be a bit of a problem using both $x_1$ and $x_2$ in a linear model, including $x_1x_2$ is unlikely to exacerbate it.

Nonconstant product

To make the calculations clearer, we might as well assume the $x_i$ have unit variance. Let the variance of $x_1x_2$ be $\tau^2$ and write $\rho_{12\cdot i}$ for the correlations between $x_1x_2$ and $x_i.$ Let's compute what happens to these correlations when constants $c_i$ are subtracted from the $x_i.$ Because the $x_i$ play perfectly symmetrical roles (just swap "$1$" for "$2$" in the indexes), it suffices to compute the correlation with $x_1:$

$$\eqalign{ \operatorname{Cor}((x_1-c_1)(x_2-c_2), x_1) &= \frac{\operatorname{Cov} ((x_1-c_1)(x_2-c_2), x_1)}{\sqrt{\operatorname{Var}{(x_1-c_1)(x_2-c_2)}\operatorname{Var}{x_1}}} \\ &= \frac{\operatorname{Cov} (x_1x_2 - c_2x_1 - c_1x_2+c_1c_2, x_1)}{\sqrt{\operatorname{Var}(x_1x_2 - c_1x_2 - c_2x_1 + c_1c_2)}} \\ &= \frac{\tau\rho_{12\cdot 1}-c_2-c_1\rho_{1\cdot 2}}{\sqrt{\tau^2 - c_1\rho_{1\cdot 2} - c_2 - 2c_1\rho_{12\cdot 2} - 2c_2\rho_{12\cdot 1} + 2c_1c_2\rho_{1\cdot 2}}}.\tag{*} }$$

Zero correlations with the product

Regardless of what the correlation between the $x_i$ might be, we can choose $(c_1,c_2)$ to make the product uncorrelated with the $x_i.$

From the foregoing analysis, this will be achieved when the numerator of $(*)$ is zero for $i=1,2:$

$$\left\{\matrix{0 = \tau\rho_{12\cdot 1} -c_2 - c_1\rho_{1\cdot 2} \\ 0 = \tau\rho_{12\cdot 2} -c_1 - c_2\rho_{1\cdot 2}}\right.$$

When $\rho_{1\cdot 2}^2 \ne 1,$ this system of equations in $(c_1,c_2)$ has a unique solution. Here, for example, is a scatterplot matrix of a dataset of $100$ values in which the $(x_i)$ have a bivariate Normal distribution with correlation $\rho_{1\cdot 2}=-0.99$ but the $x_i$ have zero correlation with $x_1x_2$:

Figure 1

Because $x_1x_2$ is uncorrelated with ("orthogonal to") both the $x_i,$ introducing it into any linear model will create no problems at all.

As this example suggests, this situation is the norm because it tends to occur when the $x_i$ have been centered. In other words, if you center your variables before creating an interaction you usually won't run into trouble with additional collinearity.

Strong correlations with the product

The equations $(*)$ can also be solved to produce strong correlations. We needn't even go so far as to solve the equations exactly (which is challenging), because there's a simple shortcut: by rescaling one of the $x_i$ to be nearly zero and adding a constant to it, we will not change their correlation, but then the product will be nearly equal to a multiple of the other one of the $x_i,$ thereby making them strongly correlated.

Here is an example based on the previous one. In this example, $x_2$ was changed to $1 + x_2 / 100$ so that $x_1x_2$ is approximately equal to $x_1,$ making it strongly positively correlated with $x_1x_2.$ Indeed, $\rho_{12\cdot 1} = 0.999878$ and $\rho_{12\cdot 2} = -0.9898793$ in this example.

Figure 2

$\endgroup$
2
  • 1
    $\begingroup$ Perfect! Thank you for the thorough explanation :) $\endgroup$
    – Hank Lin
    Dec 30, 2018 at 12:06
  • $\begingroup$ This is a superb post $\endgroup$
    – Aegis
    Dec 13, 2023 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.