19
$\begingroup$

A few days ago, a psychologist-researcher of mine told me about his method to select variables to linear regression model. I guess it's not good, but I need to ask someone else to make sure. The method is:

Look at correlation matrix between all variables (including Dependent Variable Y) and choose those predictors Xs, that correlate most with Y.

He didn't mention any criterion. Q: Was he right?

[I think that this selection method is wrong, because of many things, like it's the theory that says which predictors should be selected, or even omitted variable bias (OVB).]

$\endgroup$
5
  • $\begingroup$ I'd suggest changing title to "Is using correlation matrix to select predictors for regression correct?" or something similar to be more informative. A simple counterexample to your question is a variable that has correlation of 1 with the dependent variable - you probably won't like to use this one in your model. $\endgroup$
    – Tim
    Feb 23, 2015 at 12:12
  • 4
    $\begingroup$ There is some logic to the method, but it only works if you are restricted to select exactly one regressor. If you can select a few, this method breaks. It's because a linear combination of a few Xs that are only weakly correlated with Y may have a larger correlation with Y than a linear combination of a few Xs that are strongly correlated with Y. Recall that multiple regression is about linear combinations, not just individual effects... $\endgroup$ Feb 23, 2015 at 12:44
  • 1
    $\begingroup$ Correlation $$ \rho_{X,Y} = \frac{Cov(X, Y)}{\sigma_X \sigma_Y} $$ is just standardized regression slope $$ \hat \beta_1 = \frac{Cov(X, Y)}{\sigma_X } $$ for simple regression with one independent variable. So this approach just lets you to find the independent variable with greatest value for slope parameter, but it gets more complicated with multiple independent variables. $\endgroup$
    – Tim
    Feb 23, 2015 at 18:38
  • 2
    $\begingroup$ These replies confirm my thought about this 'method', yet many psychologists use this kind of variable selection :( $\endgroup$ Feb 24, 2015 at 8:02
  • $\begingroup$ This sounds like the 'Leekasso'. $\endgroup$ Jun 27, 2017 at 18:02

3 Answers 3

21
$\begingroup$

If, for some reason, you are going to include only one variable in your model, then selecting the predictor which has the highest correlation with $y$ has several advantages. Out of the possible regression models with only one predictor, then this model is the one with the highest standardized regression coefficient and also (since $R^2$ is the square of $r$ in a simple linear regression) the highest coefficient of determination.

But it's not clear why you would want to restrict your regression model to one predictor if you have data available for several. As mentioned in the comments, just looking at the correlations doesn't work if your model might include several variables. For example, from this scatter matrix, you might think that the predictors for $y$ you should include in your model are $x_1$ (correlation 0.824) and $x_2$ (correlation 0.782) but that $x_3$ (correlation 0.134) is not a useful predictor.

Scatter plot matrix of correlated variables

But you'd be wrong - in fact in this example, $y$ depends on two independent variables, $x_1$ and $x_3$, but not directly on $x_2$. However $x_2$ is highly correlated with $x_1$, which leads to a correlation with $y$ also. Looking at the correlation between $y$ and $x_2$ in isolation, this might suggest $x_2$ is a good predictor of $y$. But once the effects of $x_1$ are partialled out by including $x_1$ in the model, no such relationship remains.

require(MASS) #for mvrnorm 
set.seed(42) #so reproduces same result

Sigma <- matrix(c(1,0.95,0,0.95,1,0,0,0,1),3,3)
N <- 1e4
x <- mvrnorm(n=N, c(0,0,0), Sigma, empirical=TRUE)
data.df <- data.frame(x1=x[,1], x2=x[,2], x3=x[,3])
# y depends on x1 strongly and x3 weakly, but not directly on x2
data.df$y <- with(data.df, 5 + 3*x1 + 0.5*x3) + rnorm(N, sd=2)

round(cor(data.df), 3)
#       x1    x2    x3     y
# x1 1.000 0.950 0.000 0.824
# x2 0.950 1.000 0.000 0.782
# x3 0.000 0.000 1.000 0.134
# y  0.824 0.782 0.134 1.000
# Note: x1 and x2 are highly correlated
# Since y is highly correlated with x1, it is with x2 too
# y depended only weakly on x3, their correlation is much lower

pairs(~y+x1+x2+x3,data=data.df, main="Scatterplot matrix")
# produces scatter plot above

model.lm <- lm(data=data.df, y ~ x1 + x2 + x3)
summary(model.lm)

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  4.99599    0.02018 247.631   <2e-16 ***
# x1           3.03724    0.06462  47.005   <2e-16 ***
# x2          -0.02436    0.06462  -0.377    0.706    
# x3           0.49185    0.02018  24.378   <2e-16 ***

This sample size is sufficiently large to overcome multicollinearity issues in the estimation of coefficients for $x_1$ and $x_2$. The coefficient of $x_2$ is estimated near zero, and with non-significant p-value. The true coefficient is zero. The intercept and the slopes for $x_1$ and $x_3$ are estimated near their true values of 5, 3 and 0.5 respectively. Note that $x_3$ is correctly found to be a significant predictor, even though this is less than obvious from the scatter matrix.

And here is an example which is even worse:

Sigma <- matrix(c(1,0,0,0.5,0,1,0,0.5,0,0,1,0.5,0.5,0.5,0.5,1),4,4)
N <- 1e4
x <- mvrnorm(n=N, c(0,0,0,0), Sigma, empirical=TRUE)
data.df <- data.frame(x1=x[,1], x2=x[,2], x3=x[,3], x4=x[,4])
# y depends on x1, x2 and x3 but not directly on x4
data.df$y <- with(data.df, 5 + x1 + x2 + x3) + rnorm(N, sd=2)

round(cor(data.df), 3)
#       x1    x2    x3    x4     y
# x1 1.000 0.000 0.000 0.500 0.387
# x2 0.000 1.000 0.000 0.500 0.391
# x3 0.000 0.000 1.000 0.500 0.378
# x4 0.500 0.500 0.500 1.000 0.583
# y  0.387 0.391 0.378 0.583 1.000

pairs(~y+x1+x2+x3+x4,data=data.df, main="Scatterplot matrix")

model.lm <- lm(data=data.df, y ~ x1 + x2 + x3 +x4)
summary(model.lm)
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  4.98117    0.01979 251.682   <2e-16 ***
# x1           0.99874    0.02799  35.681   <2e-16 ***
# x2           1.00812    0.02799  36.016   <2e-16 ***
# x3           0.97302    0.02799  34.762   <2e-16 ***
# x4           0.06002    0.03958   1.516    0.129

Here $y$ depends on the (uncorrelated) predictors $x_1$, $x_2$ and $x_3$ - in fact the true regression slope is one for each. It does not depend on a fourth variable, $x_4$, but because of the way that variable is correlated with each of $x_1$, $x_2$ and $x_3$, it would be $x_4$ that stands out in the scatterplot and correlation matrices (its correlation with $y$ is 0.583, while the others are below 0.4). So selecting the variable with the highest correlation with $y$ can actually find the variable that does not belong in the model at all.

$\endgroup$
3
  • $\begingroup$ But ... does it all thinking apply to the situation, when this 'fellow psychologist' chooses - say - 4 of 10 variables Xs, that correlate highly with Y (correlation coefs < 0.7), leaving six other Xs that correlate moderately or not so much with Y? $\endgroup$ Feb 24, 2015 at 15:28
  • 1
    $\begingroup$ @Lili When I've got time I want to add an example to this answer, in which a variable should be included in the model which has (before we partial other variables out) zero correlation with $y$. I also suggest you look up the concepts of confounding and spurious correlation, which are certainly relevant in psychology. $\endgroup$
    – Silverfish
    Feb 24, 2015 at 16:48
  • $\begingroup$ Thank you @Silverfish . I but have a couple of doubts; 1. Is this only true for regression or it is true for classification also? 2. What if I plot a scatterplot matrix wherein I found that x1, x2, and x3 have no linear correlation among themselves, however, the correlation coefficient of x3 and y is very low, say for instance, 0.2, could I then exclude x3 from the dataset? $\endgroup$ Feb 7 at 18:11
0
$\begingroup$

You could run a step-wise regression analysis and let the software choose the variables based on F values. You could also look at Adjusted R^2 value when you run the regression each time, to see if adding any new variable contributing to your model. Your model may have the problem of multicollinearity if you just go by correlation matrix and choose variables with strong correlation. Hope this helps!

$\endgroup$
2
  • 6
    $\begingroup$ Stepwise selection leads to the same problems as method described by OP: stata.com/support/faqs/statistics/stepwise-regression-problems also note that the question was about this certain method and not about looking for alternative methods. $\endgroup$
    – Tim
    Feb 24, 2015 at 9:30
  • 2
    $\begingroup$ This is a very basic method for model selection - if your goal is strictly explanation of variance, stepwise using R2 may be appropriate, but if you are interested in inference, prediction, hypothesis testing, etc., then you need to think way beyond R2 (and perhaps even ignore R2). $\endgroup$ Feb 24, 2015 at 15:47
0
$\begingroup$

Theres nothing wrong with this method, particularly if you know about multicollinearity. Avoiding multicollinearity is very easy.

Simply steer clear of adding independent variables that correlate with one another, since using only one of said variables is necessary. If x1 and x2 both correlate with y and correlate with each other, use reasonable judgement to assess which is higher in the causal chain, and omit the latter. A strong theoretical framework can help with such a selection process.

I.e. using a correlation matrix, or even better a scatterplot matrix, can work if you know what to look for.

$\endgroup$
11
  • $\begingroup$ -1 for the comments about not needing both features when features are correlated. This is a common misconception about linear regression that partially comes from misunderstanding the Gauss-Markov theorem (I suspect). $\endgroup$
    – Dave
    Jun 24 at 0:40
  • $\begingroup$ -1 for making a comment without re-reading your answer. Correlated independent variables lead to multicolinearity. $\endgroup$ Aug 2 at 15:36
  • $\begingroup$ multicollinearity can be identified with abnormally large t-scores, typically above 10 or in some cases even as high as 20-30. Never trust a regression where correlated independent variables have t scores that are too good to be true. $\endgroup$ Aug 2 at 15:38
  • $\begingroup$ So what if there is multicollinearity? That’s not a dealbreaker. $\endgroup$
    – Dave
    Aug 2 at 15:39
  • $\begingroup$ Please see above comment $\endgroup$ Aug 2 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.