Significance of correlation and regression coefficients

Question

I have 10 variables out of which one is dependent variable and the rest are independent.

By calculating the correlation matrix between each of the dependent variable and independent variables I got some non-significant correlations but when I fitted the multiple regression model then some of the regression coefficients for which the corresponding correlation was not significant I got significant coefficient.

Why is that and is there some theory or maths behind it?

Answer 1

Let your response variable be $y$ and the independent variables be $x_i$. In the first calculation, you calculate the correlation coefficient of $(y,x_i)$. Then you calculate the multiple regression of $y$ on $x_1, x_2, \dotsc, x_9$. Then you get regression coefficients which take into account the multivariate relationships, which can be very different from the marginal correlation you first calculated. That some variable could be significant in the multiple regression, which is not correlated with $y$ marginally, is not surprising at all.

Answer 2

Very good question.

The reason simple: the coefficient $a_i$ for a variable $x_i$ in (single/multiple) regression is change in the expected value of the dependent variable $y$, for a unit change of variable $x_i$, keeping all other independent variables fixed.

That is: $a_i = \frac{\partial E[y|x_i]}{\partial x_i}$.

The specific case you are encountering, where correlation between $x_i$ and $y$ is 0, but its coefficient is statistically significant in multiple regression case can happen if you have a supressor variable: http://ericae.net/ft/tamu/supres.htm.

Here is a simple constructive example. Suppose I have three variables:

• $x \sim N(0,1)$
• $y \sim N(0,1)$
• $z = a\,x + b\,y + \epsilon$, where $a$ and $b$ are non-zero, and $\epsilon$ is some random independent noise.

Now consider the regression problem in which the predictors are $x$ and $z$, and the response variable is $y$.

Clearly, correlation between:

• $x,y$ is 0
• $x,z$ is non zero
• $y,z$ is non zero

Now, it's clear how $x$ can help improve $y$'s prediction accuracy even though it's uncorrelated with $y$. Informally, knowing $x$ helps subtract its component in $z$, which helps boost the residual signal in $z$ that can predict $y$.