# Why ANOVA/Regression results change when controlling for another variable

This question might be very basic, but somehow I don't understand this point.

Suppose initially I used a univariate regression equation such as

GDP=a+b*Income


I'll get some coefficient values (say 0.5). Now, I'm using the same structure of the regression model, but added another independent variable. So, the new equation is

GDP=a+b*Income+c*Investment


Then the new coefficients value will be b=0.3 & c=0.4.

My question is why coefficient's value changes when we add another independent variable?

Hope I can put my question clearly.

-
Because your two independent variables correlate with each other –  ttnphns Mar 31 '12 at 16:42
Thanks ttnphns! But can you please elaborate. Why correlation lead to change in coefficients? –  Ari Mar 31 '12 at 16:44

Linear regression can be illustrated geometrically in terms of an orthogonal projection of the predicted variable vector $\boldsymbol{y}$ onto the space defined by the predictor vectors $\boldsymbol{x}_{i}$. This approach is nicely explained in Wicken's book "The Geometry of Multivariate Statistics" (1994). Without loss of generality, assume centered variables. In the following diagrams, the length of a vector equals its standard deviation, and the cosine of the angle between two vectors equals their correlation (see here). The simple linear regression from $\boldsymbol{y}$ onto $\boldsymbol{x}$ then looks like this:

$\hat{\boldsymbol{y}} = b \cdot \boldsymbol{x}$ is the prediction that results from the orthogonal projection of $\boldsymbol{y}$ onto the subspace defined by $\boldsymbol{x}$. $b$ is the projection of $\boldsymbol{y}$ in subspace coordinates (basis vector $\boldsymbol{x}$). This prediction minimizes the error $\boldsymbol{e} = \boldsymbol{y} - \hat{\boldsymbol{y}}$, i.e., it finds the closest point to $\boldsymbol{y}$ in the subspace defined by $\boldsymbol{x}$ (recall that minimizing the error sum of squares means minimizing the variance of the error, i.e., its squared length). With two correlated predictors $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$, the situation looks like this:

$\boldsymbol{y}$ is projected orthogonally onto $U$, the subspace (plane) spanned by $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$. The prediction $\hat{\boldsymbol{y}} = b_{1} \cdot \boldsymbol{x}_{1} + b_{2} \cdot \boldsymbol{x}_{2}$ is this projection. $b_{1}$ and $b_{2}$ are thus the ends of the dotted lines, i.e. the coordinates of $\hat{\boldsymbol{y}}$ in subspace coordinates (basis vectors $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$).

The next thing to realize is that the orthogonal projections of $\hat{\boldsymbol{y}}$ onto $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$ are the same as the orthogonal projections of $\boldsymbol{y}$ itself onto $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$.

This allows us to directly compare the regression weights from each simple regression with the regression weights from the multiple regression:

$\hat{\boldsymbol{y}}_{1}$ and $\hat{\boldsymbol{y}}_{2}$ are the predictions from the simple regressions $\boldsymbol{y}$ onto $\boldsymbol{x}_{1}$, and $\boldsymbol{y}$ onto $\boldsymbol{x}_{2}$. Their endpoints give the individual regression weights $b^{1} = \rho_{x_{1} y} \cdot \sigma_{y}$ and $b^{2} = \rho_{x_{2} y} \cdot \sigma_{y}$, where $\rho_{x_{1} y}$ is the correlation between $\boldsymbol{x}_{1}$ and $\boldsymbol{y}$, and $\sigma_{y}$ is the standard deviation of $\boldsymbol{y}$. In contrast, the endpoints of the dotted lines give the regression weights from the multiple regression of $\boldsymbol{y}$ onto $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$: $b_{1} = \beta_{1} \sigma_{y}$, where $\beta_{1}$ is the standardized regression coefficient.

Now it is easy to see that $b^{1}$ and $b^{2}$ will coincide exactly with $b_{1}$ and $b_{2}$ only if $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$ are orthogonal (or if $\boldsymbol{y}$ is orthogonal to the plane spanned by $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$). It is also easy to geometrically construct cases that sometimes seem puzzling, e.g., when the regression weight has the opposite sign as the bivariate correlation between a predictor and the predicted variable:

Here, $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$ are highly correlated. Now the sign of the correlation between $\boldsymbol{y}$ and $\boldsymbol{x}_{1}$ is positive (red line: orthogonal projection of $\boldsymbol{y}$ onto $\boldsymbol{x}_{1}$), but the regression weight from the multiple regression is negative (end of green line onto subspace defined by $\boldsymbol{x}_{1}$.

-
+1, this is really nice. How did you make these figures? –  gung Apr 1 '12 at 12:41
This is too good. Thanks a lot for taking so much pain to answer my question. –  Ari Apr 1 '12 at 12:49
@gung Sorry, these are scans from an old text I had lying around (based on Wicken's book). –  caracal Apr 1 '12 at 14:46

Because multi-variable regression is, using economic jargon "ceteris paribus" i.e. controlling other elements unchanged.

The difference between

$GDP=\beta_{0}+\beta_{1}Income$ (1)

and $GDP=\gamma_{0}+\gamma_{1}Income+\gamma_{2}Investment$ (2)

is that $\beta_{1}$ in (1) captures the correlation (or effects) between GDP and Income and other elements (of course including Investment). That's $\beta_{1}$ absorb all effects of variable other than Income. But in (2) when Investment is added, $\gamma_{1}$ excluding the effects from Investment.

You can do a three-step regression. In the first step, regress GDP on Investment. Then you get the unexplained variance of GDP by Investment.

In the second step, regress Income on Investment. You get the unexplained variance of Income by Investment.

In the third step, regress the unexplained part of GDP by unexplained Income. Then you get $\hat{\gamma_{1}}$.

I mean $\gamma_{1}$ is direct effects of Income on GDP, excluding the indirect effects from Investment through Income on GDP.

Hope I say it clearly.

-
Thanks Yang! If I understand your answer properly, you want to say that in (1) Income is including all the direct & indirect effect. but in (2), due to the presence of Inv & c.p. condition Income is taking only the direct effect on GDP & not the indirect effect. So, that's why there's difference in coefficients. –  Ari Apr 1 '12 at 7:24