3
$\begingroup$

I have 10 variables out of which one is dependent variable and the rest are independent.

By calculating the correlation matrix between each of the dependent variable and independent variables I got some non-significant correlations but when I fitted the multiple regression model then some of the regression coefficients for which the corresponding correlation was not significant I got significant coefficient.

Why is that and is there some theory or maths behind it?

$\endgroup$
1
$\begingroup$

Let your response variable be $y$ and the independent variables be $x_i$. In the first calculation, you calculate the correlation coefficient of $(y,x_i)$. Then you calculate the multiple regression of $y$ on $x_1, x_2, \dotsc, x_9$. Then you get regression coefficients which take into account the multivariate relationships, which can be very different from the marginal correlation you first calculated. That some variable could be significant in the multiple regression, which is not correlated with $y$ marginally, is not surprising at all.

$\endgroup$
  • 1
    $\begingroup$ your suggestion of explaining this with a term marginal is logical,appealing and interesting.thanx $\endgroup$ – 李 慕 Apr 9 '16 at 20:10
  • $\begingroup$ is there a way that one could manipulate the values variables so that non-significant regression coefficients become significant ? $\endgroup$ – 李 慕 Apr 9 '16 at 23:04
  • $\begingroup$ Is it correct to use term- multivariate regression? $\endgroup$ – Subhash C. Davar Apr 11 '16 at 13:05
  • $\begingroup$ It should be multiple regression, will edit. $\endgroup$ – kjetil b halvorsen Apr 11 '16 at 14:02
  • 1
    $\begingroup$ @kjetilbhalvorsen Please have a look at my other question on moderated regression "Simulating Moderated Regression Data Set with given correlation structure" and please give me sugestion $\endgroup$ – 李 慕 Apr 13 '16 at 6:56
1
$\begingroup$

Very good question.

The reason simple: the coefficient $a_i$ for a variable $x_i$ in (single/multiple) regression is change in the expected value of the dependent variable $y$, for a unit change of variable $x_i$, keeping all other independent variables fixed.

That is: $a_i = \frac{\partial E[y|x_i]}{\partial x_i}$.

The specific case you are encountering, where correlation between $x_i$ and $y$ is 0, but its coefficient is statistically significant in multiple regression case can happen if you have a supressor variable: http://ericae.net/ft/tamu/supres.htm.

Here is a simple constructive example. Suppose I have three variables:

  • $x \sim N(0,1)$
  • $y \sim N(0,1)$
  • $z = a\,x + b\,y + \epsilon$, where $a$ and $b$ are non-zero, and $\epsilon$ is some random independent noise.

Now consider the regression problem in which the predictors are $x$ and $z$, and the response variable is $y$.

Clearly, correlation between:

  • $x,y$ is 0
  • $x,z$ is non zero
  • $y,z$ is non zero

Now, it's clear how $x$ can help improve $y$'s prediction accuracy even though it's uncorrelated with $y$. Informally, knowing $x$ helps subtract its component in $z$, which helps boost the residual signal in $z$ that can predict $y$.

$\endgroup$
  • 1
    $\begingroup$ There is also a causal/graphical model interpretation that makes this very obvious. I'll leave it to the reader as an exercise. :-) $\endgroup$ – Vimal Apr 11 '16 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.