3
$\begingroup$

Consider a polynomial regression of the form

$y = α*x_1^2 + β*x_1 + ​ɣ*x_2$

My question is: how to deal with multicollinearity between $x_1^2$ and $x2$? Or in other words, how to control for a quadratic correlation between two independent variables in polynomial regression analyses?

I guess that such correlations must have consequences for model fitting. I know that it is not perfect, but for "normal" multicollinearity in (non-polynomial) regression analyses, scientists in my research field would drop one of the correlated variables in order to avoid multicollinearity issues. Perhaps someone can indicate a sensible approach?

Here I include two example plots, $x_1$ vs $x_2$ and $x_1^2$ vs $x_2$. Hmisc::rcorr with type = "Spearman" (inR) yields a correlation coefficient between $x_1^2$ vs $x_2$ of $-0.58$. The variables are scaled variables from a use case.

enter image description here

enter image description here

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ 1) Why are $x_1^2$ and $x_2$ necessarily correlated? // What’s wrong with correlation for your modeling task? $\endgroup$
    – Dave
    Feb 24, 2021 at 13:17
  • 1
    $\begingroup$ It's not a problem. There is no need to worry about it. The purpose here is to model curvature, not to assess the effect of $x$ while holding $x^2$ fixed. $\endgroup$
    – BigBendRegion
    Feb 24, 2021 at 13:54
  • $\begingroup$ @Dave, this from a use case where I detected correlation for x1² and x2 (I don't know how to write it correctly here... sorry). // I no not know whether this is really wrong for my modeling task. An example would be: in a study, the effect of the variables x1: grazing pressure and x2: diversity of microsites on the dependent variable y: ground beetle diversity is tested. Both variables might have an effect on beetle diversity. The study sites feature highest microsite diversity at intermediate grazing pressure, which results in x1² and x2 to be correlated. I thought, this might be a problem. $\endgroup$
    – yenats
    Feb 24, 2021 at 14:27
  • $\begingroup$ @BigBendRegion, I am not sure if this is correct. I do consider more than one variable x (in the example the variables x1 and x2). I would not worry about x1 and x1² being correlated. However, may it not be a problem if x2 and x1² are correlated? $\endgroup$
    – yenats
    Feb 24, 2021 at 14:31
  • 1
    $\begingroup$ Since you wouldn't worry about correlation between $x_1$ and $x_1^2,$ why do you have any concern about correlation between $x_2$ and $x_1^2$?? $\endgroup$
    – whuber ♦
    Feb 24, 2021 at 15:01

2 Answers 2

Reset to default
1
$\begingroup$

It is to be expected that independent variables in a multivariable regression model will be correlated. Only in an experimental setting would you expect them to be orthogonal. A correlation coefficient of 0.58 is not especially large and I would not be worried about this.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for this promising answer, @Robert Long. If the correlation was higher - is there anything else you'd suggest? And do you think there is some kind of threshold beyond which one should worry about correlations between $x_1^2$ and $x_2$? $\endgroup$
    – yenats
    Mar 3, 2021 at 11:12
  • $\begingroup$ You're welcome. There is no threshold but when you are dealing with high correlations, say >0.9 and you have a lot of variables then some kind of penalised regression may be appropriate. $\endgroup$
    – Robert Long
    Mar 3, 2021 at 11:29
0
$\begingroup$

In same cases when the correlation is high you can transform your x, usually by centering it, that is making:

x* = x - mean(x)

Then the model is formulated with the transformed x*:

y=α∗x*^2+β∗x*

Of course, you need to take care when making inferences on the coefficients, but usually those do not change dramatically, but the collinearity would be very low compared to the original x. See more information here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.