3
$\begingroup$

Consider a polynomial regression of the form

$y = α*x_1^2 + β*x_1 + ​ɣ*x_2$

My question is: how to deal with multicollinearity between $x_1^2$ and $x2$? Or in other words, how to control for a quadratic correlation between two independent variables in polynomial regression analyses?

I guess that such correlations must have consequences for model fitting. I know that it is not perfect, but for "normal" multicollinearity in (non-polynomial) regression analyses, scientists in my research field would drop one of the correlated variables in order to avoid multicollinearity issues. Perhaps someone can indicate a sensible approach?

Here I include two example plots, $x_1$ vs $x_2$ and $x_1^2$ vs $x_2$. Hmisc::rcorr with type = "Spearman" (inR) yields a correlation coefficient between $x_1^2$ vs $x_2$ of $-0.58$. The variables are scaled variables from a use case.

enter image description here

enter image description here

$\endgroup$
8
  • 2
    $\begingroup$ 1) Why are $x_1^2$ and $x_2$ necessarily correlated? // What’s wrong with correlation for your modeling task? $\endgroup$
    – Dave
    Commented Feb 24, 2021 at 13:17
  • 1
    $\begingroup$ It's not a problem. There is no need to worry about it. The purpose here is to model curvature, not to assess the effect of $x$ while holding $x^2$ fixed. $\endgroup$ Commented Feb 24, 2021 at 13:54
  • $\begingroup$ @Dave, this from a use case where I detected correlation for x1² and x2 (I don't know how to write it correctly here... sorry). // I no not know whether this is really wrong for my modeling task. An example would be: in a study, the effect of the variables x1: grazing pressure and x2: diversity of microsites on the dependent variable y: ground beetle diversity is tested. Both variables might have an effect on beetle diversity. The study sites feature highest microsite diversity at intermediate grazing pressure, which results in x1² and x2 to be correlated. I thought, this might be a problem. $\endgroup$
    – yenats
    Commented Feb 24, 2021 at 14:27
  • $\begingroup$ @BigBendRegion, I am not sure if this is correct. I do consider more than one variable x (in the example the variables x1 and x2). I would not worry about x1 and x1² being correlated. However, may it not be a problem if x2 and x1² are correlated? $\endgroup$
    – yenats
    Commented Feb 24, 2021 at 14:31
  • 1
    $\begingroup$ Since you wouldn't worry about correlation between $x_1$ and $x_1^2,$ why do you have any concern about correlation between $x_2$ and $x_1^2$?? $\endgroup$
    – whuber
    Commented Feb 24, 2021 at 15:01

2 Answers 2

1
$\begingroup$

It is to be expected that independent variables in a multivariable regression model will be correlated. Only in an experimental setting would you expect them to be orthogonal. A correlation coefficient of 0.58 is not especially large and I would not be worried about this.

$\endgroup$
2
  • $\begingroup$ Thank you for this promising answer, @Robert Long. If the correlation was higher - is there anything else you'd suggest? And do you think there is some kind of threshold beyond which one should worry about correlations between $x_1^2$ and $x_2$? $\endgroup$
    – yenats
    Commented Mar 3, 2021 at 11:12
  • $\begingroup$ You're welcome. There is no threshold but when you are dealing with high correlations, say >0.9 and you have a lot of variables then some kind of penalised regression may be appropriate. $\endgroup$ Commented Mar 3, 2021 at 11:29
0
$\begingroup$

In same cases when the correlation is high you can transform your x, usually by centering it, that is making:

x* = x - mean(x)

Then the model is formulated with the transformed x*:

y=α∗x*^2+β∗x*

Of course, you need to take care when making inferences on the coefficients, but usually those do not change dramatically, but the collinearity would be very low compared to the original x. See more information here.

$\endgroup$
1
  • $\begingroup$ Note that the variable are all scaled (that is, centered and scaled). Hence, centering it again would not make much sense, would it? $\endgroup$
    – yenats
    Commented Oct 6, 2023 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.