7
$\begingroup$

I cannot differentiate clearly between "interaction" and "collinearity" in multiple linear regression. For me these terms are related but not the same.

I have searched the forum but could not find the answer. Please share your answer if you have. If you know there is an answer somewhere in the forum for this kind of question, please share the link.

Thanks

Qocande

$\endgroup$
  • $\begingroup$ Here is a source I found by googling: uk.sagepub.com/salkind2study/articles/15Article01.pdf It's not an exhaustive answer but provides some insight to the difference. $\endgroup$ – Dan Aug 29 '14 at 18:32
  • $\begingroup$ An answer suitable for beginners is at www.integrativestatistics.com/partial.htm $\endgroup$ – rolando2 Aug 31 '14 at 14:04
14
$\begingroup$

An interaction may arise when considering the relationship among three or more variables, and describes a situation in which the simultaneous influence of two variables on a third is not additive. Most commonly, interactions are considered in the context of regression analyses.

The presence of interactions can have important implications for the interpretation of statistical models. If two variables of interest interact, the relationship between each of the interacting variables and a third "dependent variable" depends on the value of the other interacting variable. In practice, this makes it more difficult to predict the consequences of changing the value of a variable, particularly if the variables it interacts with are hard to measure or difficult to control.


Collinearity is a statistical phenomenon in which two or more predictor variables in a multiple regression model are highly correlated, meaning that one can be linearly predicted from the others with a non-trivial degree of accuracy. In this situation the coefficient estimates of the multiple regression may change erratically in response to small changes in the model or the data. Collinearity does not reduce the predictive power or reliability of the model as a whole, at least within the sample data themselves; it only affects calculations regarding individual predictors. That is, a multiple regression model with correlated predictors can indicate how well the entire bundle of predictors predicts the outcome variable, but it may not give valid results about any individual predictor, or about which predictors are redundant with respect to others.


Bottom line: Interactions don't imply collinearity and collinearity does not imply there are interactions.

$\endgroup$
  • $\begingroup$ Thanks Dan. It is useful. I like the summary (i.e., the "bottom line") $\endgroup$ – qocande Aug 29 '14 at 22:25
  • 3
    $\begingroup$ (+1). Although if I add to two uncorrelated regressors $X$ and $Z$ the product $T := X \cdot Z$, then $T$ is correlated with both $X$ and $Z$. So adding interactions is often implying a certain amount of collinearity. That is what makes "main effects" somewhat unhandy to interprete. $\endgroup$ – Michael M Aug 30 '14 at 10:01
0
$\begingroup$

INTERACTION mostly applied in Two way Anova and tells the impact of two or more independent variable on a given variable i,e.each of the independent variables have the same impact on a given dependent variable.where as col-linearity indicates the correlation between two or more independent variable without including the dependent variable.

$\endgroup$
  • $\begingroup$ col-linearity is in a case of multiple regression $\endgroup$ – zerihun kinde Nov 3 '15 at 17:34
0
$\begingroup$

Interaction terms can be introduced to your model to account for different effects when two independent variables combine in some interesting way. They are commonly used when categorical factors are present, e.g. to allow different rates of response of $Y$ (income) to a second factor $X_{2}$ (years of education) according to the categorical $X_{1}$ (gender, 0=male, 1= female). If we simply regress $Y = \beta_{0} + \beta_{1}X_{1} + \beta_{2}X_{2}$, the model only accounts for females earning a fixed amount more or less than males, with a separate term accounting for educational difference regardless of gender.

If we add a third interaction variable $X_{1}X_{2}$, so $Y = \beta_{0} + \beta_{1}X_{1} + \beta_{2}X_{2} + \beta_{3}(X_{1}X_{2})$, this third factor will be zero for males, but non-zero for females, thus representing the specific variable female years of education, and allowing the model to separately account for the effects this on income ($\beta_{2}$ becomes the gradient for rate of change of male income, and $\beta_{3}$ is an adjustment to the slope of income change for females).

This is an interaction, but it is not collinear, because it does not respond the same way as years of education across the full data set (it is zero for all points that are male). Thus, interaction is not the same as collinearity, and interaction can in fact be a useful element of a regression model.

More generally, the multiplication of two variables (both non-constant) is by definition non-linear, so while an interaction term created in this way may be related to the two component variables, it is not related linearly so collinearity is impossible.

$\endgroup$
  • $\begingroup$ Please be careful about blanket assertions like "impossible." They might be correct in the simplest circumstances you envision, but they definitely are wrong in more complicated circumstances. For instance, suppose a multiple regression of $n$ observations involves categorical variables $X_1,\ldots,X_p$ having $d_1,\ldots,d_p$ levels each. The model with all pairwise interactions uses $1+\sum(d_i-1)+\sum_{j\gt i}(d_i-1)(d_j-1)$ terms, which easily can exceed $n,$ guaranteeing collinearity. $\endgroup$ – whuber Aug 5 '18 at 14:37
  • $\begingroup$ @whuber, that's interesting. I'm suggesting the individual interaction term is not linearly related to either of it's components; in the scenario you describe is this also true, and if so, what causes collinearity? (As I understand it, we should always leave out one pair to avoid this situation arising? Though that doesn't address the problem of # of terms exceeding $n$.) $\endgroup$ – James Aug 6 '18 at 1:18
  • $\begingroup$ Perhaps some insight can be gained by observing that even a single interaction between two binary variables can be a linear combination of them. Let those variables be $x$ and $y,$ coded as 0/1, and suppose that $(x,y)=(0,0)$ does not occur in the dataset. The relation $xy=x+y-1$ exhibits the interaction $xy$ as a linear combination of $x,y,$ and the constant: perfect collinearity. $\endgroup$ – whuber Aug 6 '18 at 12:23
  • $\begingroup$ @whuber, I had not considered both the "raw" factors of the constructed interaction term being categorical. I am confused.In the example you gave, some cases where $x=1$ would result in $xy=0$ and others would result in $xy=1$, right? That doesn't seem to me like $x$ and $xy$ are linearly related ... ? I guess it is because we would regress to the mean of the 0 and 1, and since we have only two $x$ values to plot, we get a straight line? Or are we meant to be looking at the 3-dimensional linearity of $x, y$ and $xy$ ? (I realise I'm likely completely confused by now and talking nonsense.) $\endgroup$ – James Aug 9 '18 at 10:01
  • $\begingroup$ Your comment doesn't seem to have much to do with the sense of "linear" in linear algebra or linear modeling, so perhaps you are using this term in some other sense? $\endgroup$ – whuber Aug 9 '18 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.