This question might be very basic, but somehow I don't understand this point.

Suppose initially I used a univariate regression equation such as


I'll get some coefficient values (say 0.5). Now, I'm using the same structure of the regression model, but added another independent variable. So, the new equation is


Then the new coefficients value will be b=0.3 & c=0.4.

My question is why coefficient's value changes when we add another independent variable?

Hope I can put my question clearly.

  • $\begingroup$ Because your two independent variables correlate with each other $\endgroup$ – ttnphns Mar 31 '12 at 16:42
  • $\begingroup$ Thanks ttnphns! But can you please elaborate. Why correlation lead to change in coefficients? $\endgroup$ – Beta Mar 31 '12 at 16:44

Linear regression can be illustrated geometrically in terms of an orthogonal projection of the predicted variable vector $\boldsymbol{y}$ onto the space defined by the predictor vectors $\boldsymbol{x}_{i}$. This approach is nicely explained in Wicken's book "The Geometry of Multivariate Statistics" (1994). Without loss of generality, assume centered variables. In the following diagrams, the length of a vector equals its standard deviation, and the cosine of the angle between two vectors equals their correlation (see here). The simple linear regression from $\boldsymbol{y}$ onto $\boldsymbol{x}$ then looks like this:

enter image description here

$\hat{\boldsymbol{y}} = b \cdot \boldsymbol{x}$ is the prediction that results from the orthogonal projection of $\boldsymbol{y}$ onto the subspace defined by $\boldsymbol{x}$. $b$ is the projection of $\boldsymbol{y}$ in subspace coordinates (basis vector $\boldsymbol{x}$). This prediction minimizes the error $\boldsymbol{e} = \boldsymbol{y} - \hat{\boldsymbol{y}}$, i.e., it finds the closest point to $\boldsymbol{y}$ in the subspace defined by $\boldsymbol{x}$ (recall that minimizing the error sum of squares means minimizing the variance of the error, i.e., its squared length). With two correlated predictors $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$, the situation looks like this:

enter image description here

$\boldsymbol{y}$ is projected orthogonally onto $U$, the subspace (plane) spanned by $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$. The prediction $\hat{\boldsymbol{y}} = b_{1} \cdot \boldsymbol{x}_{1} + b_{2} \cdot \boldsymbol{x}_{2}$ is this projection. $b_{1}$ and $b_{2}$ are thus the ends of the dotted lines, i.e. the coordinates of $\hat{\boldsymbol{y}}$ in subspace coordinates (basis vectors $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$).

The next thing to realize is that the orthogonal projections of $\hat{\boldsymbol{y}}$ onto $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$ are the same as the orthogonal projections of $\boldsymbol{y}$ itself onto $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$.

enter image description here

This allows us to directly compare the regression weights from each simple regression with the regression weights from the multiple regression:

enter image description here

$\hat{\boldsymbol{y}}_{1}$ and $\hat{\boldsymbol{y}}_{2}$ are the predictions from the simple regressions $\boldsymbol{y}$ onto $\boldsymbol{x}_{1}$, and $\boldsymbol{y}$ onto $\boldsymbol{x}_{2}$. Their endpoints give the individual regression weights $b^{1} = \rho_{x_{1} y} \cdot \sigma_{y}$ and $b^{2} = \rho_{x_{2} y} \cdot \sigma_{y}$, where $\rho_{x_{1} y}$ is the correlation between $\boldsymbol{x}_{1}$ and $\boldsymbol{y}$, and $\sigma_{y}$ is the standard deviation of $\boldsymbol{y}$. In contrast, the endpoints of the dotted lines give the regression weights from the multiple regression of $\boldsymbol{y}$ onto $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$: $b_{1} = \beta_{1} \sigma_{y}$, where $\beta_{1}$ is the standardized regression coefficient.

Now it is easy to see that $b^{1}$ and $b^{2}$ will coincide exactly with $b_{1}$ and $b_{2}$ only if $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$ are orthogonal (or if $\boldsymbol{y}$ is orthogonal to the plane spanned by $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$). It is also easy to geometrically construct cases that sometimes seem puzzling, e.g., when the regression weight has the opposite sign as the bivariate correlation between a predictor and the predicted variable:

enter image description here

Here, $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$ are highly correlated. Now the sign of the correlation between $\boldsymbol{y}$ and $\boldsymbol{x}_{1}$ is positive (red line: orthogonal projection of $\boldsymbol{y}$ onto $\boldsymbol{x}_{1}$), but the regression weight from the multiple regression is negative (end of green line onto subspace defined by $\boldsymbol{x}_{1}$.

  • $\begingroup$ @gung Sorry, these are scans from an old text I had lying around (based on Wicken's book). $\endgroup$ – caracal Apr 1 '12 at 14:46

Because multi-variable regression is, using economic jargon "ceteris paribus" i.e. controlling other elements unchanged.

The difference between

$GDP=\beta_{0}+\beta_{1}Income$ (1)

and $GDP=\gamma_{0}+\gamma_{1}Income+\gamma_{2}Investment$ (2)

is that $\beta_{1}$ in (1) captures the correlation (or effects) between GDP and Income and other elements (of course including Investment). That's $\beta_{1}$ absorb all effects of variable other than Income. But in (2) when Investment is added, $\gamma_{1}$ excluding the effects from Investment.

You can do a three-step regression. In the first step, regress GDP on Investment. Then you get the unexplained variance of GDP by Investment.

In the second step, regress Income on Investment. You get the unexplained variance of Income by Investment.

In the third step, regress the unexplained part of GDP by unexplained Income. Then you get $\hat{\gamma_{1}}$.

I mean $\gamma_{1}$ is direct effects of Income on GDP, excluding the indirect effects from Investment through Income on GDP.

Hope I say it clearly.

  • $\begingroup$ Thanks Yang! If I understand your answer properly, you want to say that in (1) Income is including all the direct & indirect effect. but in (2), due to the presence of Inv & c.p. condition Income is taking only the direct effect on GDP & not the indirect effect. So, that's why there's difference in coefficients. $\endgroup$ – Beta Apr 1 '12 at 7:24

The important thing to understand about regression is that you are finding estimated parameter values that minimize the sum of squared residuals. Adding (or subtracting!) covariates to your model will always change the parameter estimates unless the new covariate is perfectly orthogonal to those already in the model, or perfectly orthogonal to the response variable, or both. Furthermore, the issue isn't whether these variables are related in the population, but rather in your sample; it is quite reasonable to imagine that some variables aren't actually related to each other, but when you gather a sample, they will pretty much never be perfectly orthogonal in your sample.

  • $\begingroup$ Thanks Gung for your reply. My problem in understanding this issue is how come "correlation" between 2 independent variable lead to change in coefficients. I guess Yang to great extend help me understand this issue. But I'm not confident yet . $\endgroup$ – Beta Apr 1 '12 at 7:20

This is also called the Regression Anatomy. Regressing GDP on Income and Investment gives you the same coefficient on Income as this two-step procedure: first regress Income on Investment and predict the residuals; then regress GDP on these residuals. The residuals have the property that they only contain that part of Income that is uncorrelated with Investment. Unless the two explanatory variables are uncorrelated the multivariate coefficient is different from the bivariate coefficient. As stated above, the multivariate regression makes sure that only the direct effect of Income is captured. Hope this helps Michael


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.