Suppose we have a multiple regression with $n$ predictors. All of them have significant correlations with the response except for one. Could the predictor with the zero correlation to the response be significant?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ With "zero correlation", do you mean this is true theoretical in the underlying distribution, or do you mean this is true in the actual sample? $\endgroup$– coffeinjunkyCommented Nov 9, 2016 at 0:25
-
$\begingroup$ Actually, both cases would be interesting. I guess if it's only true in the sample, the predictor could still be significant due to some sampling error. $\endgroup$– mssCommented Nov 9, 2016 at 0:29
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
The prototypical situation in which a variable has a zero marginal correlation with the response, but a significantly non-zero association conditional on the inclusion of the other covariates is called suppression. There are a number of threads on this topic on CV:
- Suppression effect in regression: definition and visual explanation/depiction
- X and Y are not correlated, but X is significant predictor of Y in multiple regression. What does it mean?
- How can including an IV uncorrelated with the DV improve a multiple regression model?
It is also possible that the variable is not a suppressor, but is relevant and is just 'cancelled out' by another variable with the opposite effect that it is correlated with. Here are some things to read to help understand that:
answered Nov 9, 2016 at 13:32
-
$\begingroup$ very interesting, now I understand that a suppressor could have no explanatory power on response, but it can strengthen the effect of another predictor. is it possible to pair up or group predictors so that it would be easy to see what predictors are getting amplified by the suppressor? $\endgroup$– mssCommented Nov 9, 2016 at 17:35
-
$\begingroup$ That's a good question, @mss. I'll have to think about it. Generally, you have a specific hypothesis a-priori that you are assessing (or in a predictive model, you just don't care); you don't usually throw in a bunch of variables & try to figure out which is which. Among other things, there are multiplicity issues that way. You could ask a new question about this. $\endgroup$ Commented Nov 9, 2016 at 17:59
-
-
$\begingroup$ in my particular case it's more of an exploratory analysis and a lot of IVs were thrown into regression. there is an intuitive reason behind each IV. However, it feels like some of them are not necessary or redundant. I'm trying to see if there is a sound method to drop some IVs. $\endgroup$– mssCommented Nov 10, 2016 at 2:44
-
$\begingroup$ @mss, that's a question of feature selection, not primarily about understanding suppression. Feature selection is a very tough issue. Although written in a different context, you might be interested in reading my answer here. Your best bet for feature selection is to replicate your study or have hold out data in some form. $\endgroup$ Commented Nov 10, 2016 at 3:13
$\begingroup$
$\endgroup$
2
Yes, it can happen and even more extreme scenarios might happen.
Think that you have 2 features, the beginning of a period $s$ and its end $e$. Assume the the concept is the period length $l = (e-s)$ and we will build the data set so both $s$ and $e$ are independent of $l$.
Correlation consider only a single feature and the concept and it will be $0$ (by construction). However, given the start and the end of the period, you have all the information on its length.
Returning to your question, you can build a scenario in which the zero correlation predictor and on of the others predictor sum is the concept. Multiple regression is very suitable to these problems so you can get perfect prediction. That will be even easier if you'll remove all the rest of the predictors.
-
$\begingroup$ I was trying to see if somehow using the correlation matrix I can do variable selection. Your simple example clearly shows that won't work. I have a regression with a large number of predictors and I'm very surprised to see that almost all of them are significant. I know that the t-stats could be biased upward for many reasons. However, assuming there no biases, what are the ways to determine if all the predictors are truly needed in the regression? $\endgroup$– mssCommented Nov 9, 2016 at 13:02
-
1$\begingroup$ You are trying to solve a feature selection problem en.wikipedia.org/wiki/Feature_selection There are plenty of methods to cope with the problem (e.g., see here in the tag and at Wikipedia). However, please not that this is an NP-complete problem and we don't have a fast algorithm to the problem. $\endgroup$– DaLCommented Nov 9, 2016 at 13:14