Before fitting a multivariable regression model it's common to check if the predictors are correlated.
That can be done viewing the correlation matrix, at least for linear effects.

Simple least squares regression needs that the predictor variables are independent.
We could tolerate small correlations but the problem gets serious if the variables are perfectly collinear.

It's common to drop some of the correlated variables, keeping the most meaningful.
More complex alternative methods such as PCA o Ridge regression exist.

But here is my question:
If your model include interactions that interactions use to be very correlated with other variables.
For example in $F=a+b·X+c·Y+d·X·Y$
$X·Y$ is likely to be very correlated with $X$ and $Y$.

If my model has an interaction term (and it's statistically significant and important for me) but it's very correlated with some variables...
Should I drop it or keep it?
If I keep it I will be violating the regression condition of non-correlated terms.

  • $\begingroup$ You cant have Y on both sides of the equation, you are predicting something by itself? Please consider clarifying $\endgroup$ – Repmat Aug 1 '16 at 12:35

Keep it. It's one of those choices between unbiasedness and precision.

The negative effect of having correlated independent variables is that they inflate each other's variance (the statistics that quantifies this phenomenon is called variance inflation factor). The results are enlarged standard error, which leads to lower t-statistics, which lead to higher p-value.

If the interaction term is already statistically significant, then the above problem does not concern your model as much. Yet on the other hand, taking it out can have drastic consequence because without that interaction the estimates in the model ($b$ and $c$) can be biased. Once an estimate is biased there is really not much of a point to discuss its precision. For that reason it's better keep the interaction term.

Also, I'd suggest careful consideration when checking possible interactions. While we can statistically examine each of them, the importance of having a causal framework behind cannot be emphasized enough. Lastly, many analyses were not powered for checking interactions (most researchers didn't consider this as a formal hypothesis) so be mindful not to fool yourself into thinking there isn't one if there isn't one.

  • $\begingroup$ Hypothesis testing is still highly relevant even if used when the estimators are biased. Especially if there is a case to made for consistency. $\endgroup$ – Repmat Aug 1 '16 at 12:56
  • $\begingroup$ @Repmat, if it's like measurement error, where everyone is $x$ units off and yet we can still figure out differences between, say, two sexes then I agree. If it's like we found slight benefit of a drug but in fact most of them goes to people without heart problem and some harm goes to people with heart disease then I don't agree. In the latter case there isn't much to gain from consistently recovering the wrong estimate. $\endgroup$ – Penguin_Knight Aug 1 '16 at 13:04
  • $\begingroup$ Consistency can recover the right estimate regardless of any bias - that's my point. An unbiased estimator does not guarantee that the point estimate will be remotely close to the true effect, in fact it the difference can be arbitrarily large. $\endgroup$ – Repmat Aug 1 '16 at 16:40
  • $\begingroup$ @Repmat, perhaps it's lingo difference. I was trained in epidemiology and did not share the same concepts you stated. If you do find any glaring mistakes in my response please feel free to edit it. Thanks. $\endgroup$ – Penguin_Knight Aug 1 '16 at 17:03
  • $\begingroup$ Then it affects other's variance (and therefore their p-values) but not the estimation of the parameters? IMO the latter would be much worse. But when we have perfect multicollinearity the regression can become unsolvable if we don't remove one variable. I'm happy to see my question is not downvoted, I was afraid of asking trivial or wrong things. $\endgroup$ – skan Aug 1 '16 at 21:37

Consider (approximately) centreing your $X$ and $Y$ variables which will minimise the correlation between them and their interaction.

Although it is true that interpretation of the results of a regression can be easier if the predictors are independent it is not essential. After all the reason we do it is to see the effect of each predictor over and above the effect of the others.

  • $\begingroup$ And one more question. In the correlation matrix... What's the meaning of the correlation between the Intercept and any coefficient?. What should I do if one of that correlations is close to 1? $\endgroup$ – skan Oct 31 '16 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.