Do variables with similar correlation coefficient values multicollinear?

Question

For example I have two variables, X1 and X2, with which I calculate the Pearson Correlation Coefficients with a target Y. Say, if both of them result in 0.6 as the correlation coefficient, can I say X1 and X2 are multicollinear so I can only keep one of them in my linear regression model? Thanks.

Answer 1

If I am understanding you correctly, you say that $cor(X_1,y)=0.6$ and $cor(X_2,y)=0.6$. That does not mean that your model is facing collinearity. Collinearity is meassured as a large correlation between predictive variables in your model. Following your example, you would be facing collinearity if $cor(X_1, X_2)=0.9$.

Intuitively

Imagine that this is the case and $cor(X_1, X_2)=0.9$. Intuitively, the problem with it is that the information that $X_1$ privides overlaps with the information that $X_2$ provides. Both of them provide roughtly the same information so you can remove one of the variables.

Mathematically

Mathematically, the analytical solution of a linear regression model is

$$\hat\beta = (X^tX)^{-1}X^ty$$

If $cor(X_1, X_2)=0.9$ this means that $X_1$ is a linear function of $X_2$ and thus $(X^tX)$ is not invertible and the model cannot be solved (or, if solved, the solution is not stable, small changes in the data can lead to large changes in the coefficients$

Some code

Lets generate a fake dataset using R programming language

x = seq(1, 100)
y = x + rnorm(100, sd=4)
summary(lm(y~x))

Call:
  lm(formula = y ~ x)

Residuals:
  Min      1Q  Median      3Q     Max 
-9.2178 -2.0213 -0.3459  2.2776  9.1166 

Coefficients:
  Estimate Std. Error t value Pr(>|t|)    
(Intercept) -1.19308    0.69892  -1.707    0.091 .  
x            1.03636    0.01202  86.252   <2e-16 ***
  ---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.468 on 98 degrees of freedom
Multiple R-squared:  0.987, Adjusted R-squared:  0.9869 
F-statistic:  7439 on 1 and 98 DF,  p-value: < 2.2e-16

Observe here that the p-value associated to variable $x$ is < 2.2e-16. This means that it is smaller than 2.2e-16 (basically, that it is 0). Also, observe that the $R^2$ coefficient is 0.98, so with variable $x$ we explain pretty much all the behavior of $y$. Now I generate a new variable x2 that has a large correlation with x,

x2 = x
x2[1] = 3
cor(x, x2)
0.9999769

So $cor(X, X_2)=0.99$. And I solve a new model with the two variables:

summary(lm(y~x+x2))

    Call:
  lm(formula = y ~ x + x2)

Residuals:
  Min      1Q  Median      3Q     Max 
-9.2253 -2.0085 -0.3042  2.3064  9.1034 

Coefficients:
  Estimate Std. Error t value Pr(>|t|)  
(Intercept) -1.26943    0.71571  -1.774   0.0793 .
x            0.08308    1.77381   0.047   0.9627  
x2           0.95442    1.77588   0.537   0.5922  
---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.481 on 97 degrees of freedom
Multiple R-squared:  0.987, Adjusted R-squared:  0.9868 
F-statistic:  3693 on 2 and 97 DF,  p-value: < 2.2e-16

Observe that here both variables $x$ and $x_2$ have large p-values, this means that none of them is considered as a significant variable for our model. But still, the $R^2$ coefficient is large (0.98). This is indicating that the second model has some colinearity problems. Since both variables $x$ and $x_2$ explain the same, none of them is correctly identified as significant, but globally, our model is able to explain the behavior of $y$.

Answer 2

Collinearity is meassured as a large correlation between predictive variables in your model.

I just wanted to add something to this statement.

Although multicollinearity might be a problem that is also determined by calculation. It is also determined by the research objective and the research field. Having multiple items in a SEM would lead to the ultimate assumption that we have not clearly defined our items, thus even smaller thresholds might be feasible. There is no straight line, you can cross to say we have multicollinearity. Although higher values are clearly outlined, a coll. of 0.5 to 0.6 could also be problematic.

Researchers in germany and in other countries state different thresholds for having multicollinearity:

- Urban/Mayerl 2014, Structural Eqution Modeling (on german) highlights coll > 0.8

- Hair et al. 2010, Multivariate data analysis, who proposes a freaky 0.10 as a threshold, p. 204 f.*

I would advise the OP Cohen et al. 2003 Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, p. 420 to have a look on this subject.

This discussion is also extended to the values of the VIF which can reflect some of this phenomenon: https://www.researchgate.net/post/Multicollinearity_issues_is_a_value_less_than_10_acceptable_for_VIF