I have been running a few linear regression models to test the absolute and relative effect of several independent variables related to spending/investment on different tools on one measure of performance that I want to increase. I first ranked the Pearson correlation of these independent variables/tools, with some having high correlation and some low ones. Then I ran a linear regression with the dependent and independent variables, it end up that most of ones with high Pearson's also have high coefficient (most of the time also high t-stat) in regression. However, a small number of tools have very high Pearson's but extremely low or non-existent coefficient, meaning their effect is almost not there to be seen - what can be some technical/statistical/mathematical explanation for it?

  • $\begingroup$ Are you describing several different single-variable regressions or are you describing one regression with all the variables? Also are you getting the correlations independently? I am not clear from your description. $\endgroup$ Dec 9 '16 at 19:39

They are correlated with other variables in the model. When you correlate those variables (say, $x$) with the response ($y$), you are looking at them in isolation. Thus, you are measuring their association, plus the association of all the variables they are correlated with (e.g., $z$). It can be the case that there is no actual correlation between the variable and the response ($r_{x,y}=0$), but high correlations between $z$ and $y$, and between $x$ and $z$. So when you run the univariate correlation, $x$ just acts as a proxy for $z$ and you get a high univariate correlation, but when you include $z$, you see that $x$ was irrelevant.

For more information, see my answer here: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

  • $\begingroup$ gung is using one way to explain the muticollinearity problem in multiple regression. But I think as Doctor/Ambient says there are things about your approach that need to be explained. Are the independent variables being correlated with the dependent variables or are the dependent variables correlated with each other? This all relates to whether or not you are doing one variable at a time in the regression models or several. Why are you ranking the correlations? $\endgroup$ Dec 9 '16 at 20:18
  • $\begingroup$ I am inclined to think that an approach, assuming the poster is talking about separate regression models for each $x$, is to look at the formula: $\beta = r_{x,y} \cdot \frac{s_y}{s_x}$ and consider various values for $s_x$, but I am not sure if the definition of $r$ prohibits this by locking that value in some way. That might allow addressing the specific question asked, I think. $\endgroup$ Dec 9 '16 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.