3
$\begingroup$

Suppose we have a multiple regression with $n$ predictors. All of them have significant correlations with the response except for one. Could the predictor with the zero correlation to the response be significant?

$\endgroup$
  • $\begingroup$ With "zero correlation", do you mean this is true theoretical in the underlying distribution, or do you mean this is true in the actual sample? $\endgroup$ – coffeinjunky Nov 9 '16 at 0:25
  • $\begingroup$ Actually, both cases would be interesting. I guess if it's only true in the sample, the predictor could still be significant due to some sampling error. $\endgroup$ – mss Nov 9 '16 at 0:29
6
$\begingroup$

The prototypical situation in which a variable has a zero marginal correlation with the response, but a significantly non-zero association conditional on the inclusion of the other covariates is called suppression. There are a number of threads on this topic on CV:

It is also possible that the variable is not a suppressor, but is relevant and is just 'cancelled out' by another variable with the opposite effect that it is correlated with. Here are some things to read to help understand that:

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ very interesting, now I understand that a suppressor could have no explanatory power on response, but it can strengthen the effect of another predictor. is it possible to pair up or group predictors so that it would be easy to see what predictors are getting amplified by the suppressor? $\endgroup$ – mss Nov 9 '16 at 17:35
  • $\begingroup$ That's a good question, @mss. I'll have to think about it. Generally, you have a specific hypothesis a-priori that you are assessing (or in a predictive model, you just don't care); you don't usually throw in a bunch of variables & try to figure out which is which. Among other things, there are multiplicity issues that way. You could ask a new question about this. $\endgroup$ – gung - Reinstate Monica Nov 9 '16 at 17:59
  • $\begingroup$ here is a follow up question $\endgroup$ – mss Nov 10 '16 at 2:44
  • $\begingroup$ in my particular case it's more of an exploratory analysis and a lot of IVs were thrown into regression. there is an intuitive reason behind each IV. However, it feels like some of them are not necessary or redundant. I'm trying to see if there is a sound method to drop some IVs. $\endgroup$ – mss Nov 10 '16 at 2:44
  • $\begingroup$ @mss, that's a question of feature selection, not primarily about understanding suppression. Feature selection is a very tough issue. Although written in a different context, you might be interested in reading my answer here. Your best bet for feature selection is to replicate your study or have hold out data in some form. $\endgroup$ – gung - Reinstate Monica Nov 10 '16 at 3:13
3
$\begingroup$

Yes, it can happen and even more extreme scenarios might happen.

Think that you have 2 features, the beginning of a period $s$ and its end $e$. Assume the the concept is the period length $l = (e-s)$ and we will build the data set so both $s$ and $e$ are independent of $l$.

Correlation consider only a single feature and the concept and it will be $0$ (by construction). However, given the start and the end of the period, you have all the information on its length.

Returning to your question, you can build a scenario in which the zero correlation predictor and on of the others predictor sum is the concept. Multiple regression is very suitable to these problems so you can get perfect prediction. That will be even easier if you'll remove all the rest of the predictors.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I was trying to see if somehow using the correlation matrix I can do variable selection. Your simple example clearly shows that won't work. I have a regression with a large number of predictors and I'm very surprised to see that almost all of them are significant. I know that the t-stats could be biased upward for many reasons. However, assuming there no biases, what are the ways to determine if all the predictors are truly needed in the regression? $\endgroup$ – mss Nov 9 '16 at 13:02
  • 1
    $\begingroup$ You are trying to solve a feature selection problem en.wikipedia.org/wiki/Feature_selection There are plenty of methods to cope with the problem (e.g., see here in the tag and at Wikipedia). However, please not that this is an NP-complete problem and we don't have a fast algorithm to the problem. $\endgroup$ – DaL Nov 9 '16 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.