5
$\begingroup$

If I have some number of independent variables, and one dependent variable, and some of those independent variables are strongly correlated with each other, does that make one of them redundant?

$\endgroup$
3
  • 1
    $\begingroup$ Yes, it can. This phenomenon is known as multicollinearity. You can learn the basic about it in this wiki page, and more about the problems and detection in these websites (1, 2) $\endgroup$ Oct 17 '18 at 18:27
  • 2
    $\begingroup$ Although @Penguin is correct, I think it is important to show how the opposite conclusion can apply, because many well-known techniques (such as principal component regression) ignore that possibility altogether. $\endgroup$
    – whuber
    Oct 17 '18 at 19:08
  • $\begingroup$ Excellent answer and detailed explanation by whuber. In my opinion, correlated independent variables do not make each other redundant, they can improve the accuracy of predictions as demonstrated in whuber's answer. However, keeping correlated variables in model can seriously hamper the interpretation, i.e. inference of the model. $\endgroup$ Oct 19 '18 at 13:45
6
$\begingroup$

Not necessarily. It is instructive to understand why not.

The issue is whether some linear combination of the variables is linearly correlated with the response. Sometimes a set of explanatory variables can be extremely closely correlated, but removing any single one of those variables significantly reduces the quality of the model.

This can be illustrated through a simulation. The R code below does the following:

  1. It creates $n$ independent realizations of two explanatory variables $X_1$ and $X_2$ randomly in the form $$X_1=Z + \rho E,\ X_2 = Z-\rho E$$ where $Z$ and $E$ are independent standard Normal variables and $|\rho|$ is intended to be a small number. Since the variance of each $X_i$ is $$\operatorname{Var}(X_i) = \operatorname{Var}(Z \pm \rho E) = 1 \pm 2(0) + (\pm\rho)^2 = 1+\rho^2,$$ the correlation of the $X_i$ is therefore $$\operatorname{Cor}(X_1,X_2) = \frac{\operatorname{Cov}(Z+\rho E, Z-\rho E)}{1+\rho^2} = \frac{1-\rho^2}{1+\rho^2} \approx 1 - 2\rho^2.$$ For smallish $\rho$ that's very strong correlation.

  2. It realizes $n$ responses from the random variable $$Y = E + \sigma W$$ where $W$ is another Standard normal variable independent of $Z$ and $E.$ Algebra shows $$Y = \frac{1}{2\rho}X_1 + \frac{-1}{2\rho}X_2 + \sigma W.$$ When $\sigma/\rho$ is not too small, these coefficients of the $X_i$ are large relative to the random error $\sigma W.$ That makes this a very strong linear dependence of $Y$ on the $X_i.$

  3. It fits three ordinary least squares models,

    • $E[Y] = \alpha_0 + \alpha_1 X_1 + \alpha_2 X_2$ (a multiple regression),
    • $E[Y] = \beta_0 + \beta_1 X_1$ (an ordinary regression against $X_1$), and
    • $E[Y] = \gamma_0 + \gamma_1 X_2$ (an ordinary regression against $X_2$).

$Z$ plays the role of a "nuisance:" it represents a great deal of variability common to all regressors but unrelated to the response.

Let's study this situation. Here is a scatterplot showing an example of the data the simulation produces (for a sample size of $24$) when $\rho=0.1$ and $\sigma=0.01.$

Figure

The strong positive correlation of the $X_i$ is apparent, even in this small sample, through the clustering of data along the diagonals of the $X_1-X_2$ scatterplots. $Y$ doesn't appear to have much of a linear relationship to the $X_i,$ as the random scatter in the top and left plots and the results of the two ordinary regressions show.

The ordinary regression results are similar, so I will display only the first one.

Call:
lm(formula = y ~ x1)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.142002 -0.059922 -0.004332  0.048009  0.137539 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)  
(Intercept)  0.041016   0.016586   2.473   0.0216 *
x1          -0.007055   0.014991  -0.471   0.6426  
---
F-statistic: 0.2215 on 1 and 22 DF,  p-value: 0.6426

I would like to draw your attention to (1) the insignificant p-value of $0.64,$ (2) the estimated coefficient of nearly $0,$ and (3) that the residuals are typically around $0.05$ in size.

Compare this to the multiple regression using both variables:

Call:
lm(formula = y ~ x1 + x2)

Residuals:
       Min         1Q     Median         3Q        Max 
-0.0186887 -0.0078369  0.0002194  0.0039288  0.0214359 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.002080   0.002443   0.851    0.404    
x1           0.498225   0.014422  34.547   <2e-16 ***
x2          -0.496415   0.014036 -35.368   <2e-16 ***
---
F-statistic: 631.8 on 2 and 21 DF,  p-value: < 2.2e-16

This time,

  1. The p-value is essentially zero: this is a "highly significant" linear response.

  2. The estimated coefficients, close to $\pm 1/2,$ are sizable (much larger than the estimates in the single-regressor situations), close to what we know the true values of $\alpha_1$ and $\alpha_2$ to be, and both are highly significant.

  3. The residuals are typically around $0.01$ or less in size--almost an order of magnitude reduction. In most applications this would represent a huge improvement in the regression.

These are all dramatic differences, showing why even when regressors are correlated, it might be useful to include them all.

It can be instructive to play with this simulation by modifying its parameters $n,$ $\rho,$ and $\sigma.$ (Remove the set.seed line in order to get varying results.)

n <- 24
rho <- 0.1
sigma <- 0.01

set.seed(17)
eps <- rnorm(n) * rho
x1 <- rnorm(n)
x2 <- x1 - eps
x1 <- x1 + eps
y <- eps + rnorm(n) * sigma
pairs(cbind(y, x1, x2), pch=21, bg="#00000020")

summary(lm(y ~ x1 + x2))

summary(lm(y ~ x1))

summary(lm(y ~ x2))
$\endgroup$
3
  • 1
    $\begingroup$ This is a very interesting example, thanks for showing us that! I did also have a second thought as the OP actually tagged "machine-learning," I was wondering in terms of prediction, would it actually be fine to include correlated independent variables? As sometimes I saw neither of the predictors x1, or x2 is significant due to collinearity, but the overall F-test of the model is significant. $\endgroup$ Oct 17 '18 at 19:22
  • $\begingroup$ Thank you for the detailed response! To add to the above, does the above still hold when we know the correlated predictors to themselves be highly correlated with the dependent variable? $\endgroup$ Oct 17 '18 at 19:25
  • $\begingroup$ There are many variations worth considering. Change the code from y <- eps + rnorm(n) * sigma to y <- eps + x1 + rnorm(n) * sigma to study this case. The scatterplot matrix will testify that now all three mutual correlations are high. Look especially at the residuals of the three models. $\endgroup$
    – whuber
    Oct 17 '18 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.