1
$\begingroup$

Say I have a regression with 3 independent variables and I decide to introduce a 4th variable and rerun the regression.

A previous post states that the coefficient on an original variable will change when the new variable is correlated with that original variable AND when the new variable is correlated with the response. (Does adding more variables into a multivariable regression change coefficients of existing variables?)

How do we show that mathematically?

What I know: Let's say the original model was assumed to be $E[Y]=X\beta$ and the new model is assumed to be $E[Y]=X\beta + Z\gamma$, where the new data matrix $(X,Z)$ has full rank. Then the new vector of OLS coefficient estimates is:

$$\begin{bmatrix}\hat{\gamma}\\\hat{\beta}_{NEW}\end{bmatrix}=\begin{bmatrix}(Z^T(I-P_x)Z)^{-1}Z^T(I-P_x)Y\\\hat{\beta}_{OLD}-(X^TX)^{-1}X^TZ\hat{\gamma}\end{bmatrix}$$

where $I-P_x=I-X(X^TX)^{-1}X^T$ is the projection matrix onto the subspace orthogonal to the column space of X. This implies that the original coefficients will not change when either:

  1. The columns of X are orthogonal to the columns of Z ($X^TZ=0$). When there is an intercept, this implies the columns of $Z$ are centered and therefore the columns of X and Z are uncorrelated.
  2. The columns of Z are orthogonal to the residuals from the regression of Y on X. ($Z^T(I-P_x)Y=0$)

But how to you get to the answer linked above?

$\endgroup$
  • $\begingroup$ There is an incredibly simple geometrical proof: because the new variable is orthogonal to the space spanned by the previous variables and the response, its introduction changes nothing, QED. See stats.stackexchange.com/a/113207/919 for pictures. For algebraic approaches you can use Gram-Schmidt orthogonalization; for more about this, see stats.stackexchange.com/a/46508/919, which shows how to reduce the problem from three variables to just one. $\endgroup$ – whuber Jan 26 at 20:28
  • $\begingroup$ This is very helpful in gaining some intuition about the two conditions I gave, thank you! However, I'm still not clear on how gung's answer in the linked post can be explained by either the geometry you present or the conditions I have given. For example, how do you explain the case when a new variable (a column of $Z$, call it $Z_j$) is not orthogonal, but is uncorrelated to an original variable (a column of $X$, call it $X_i$). gung's answer says the coefficient on $X_i$ would not change in that case. My conditions (and I believe the geometry you present) do not speak to this case. $\endgroup$ – posteriormean Jan 28 at 9:59
  • $\begingroup$ Geometrically, in a Euclidean space (or even a Hilbert space for that matter) the orthogonal projection of a vector $Y$ on $X$ does not depend on any vectors that might be orthogonal to $X.$ Because at most three dimensions are involved here, you can literally draw a picture. The algebra (if you need it) will echo this intuition. $\endgroup$ – whuber Jan 28 at 13:52
  • $\begingroup$ But what about when the new variable is NOT orthogonal but is uncorrelated to a variable already in the regression? What math tells us that the original variable's coefficient will not change in this case? $\endgroup$ – posteriormean Jan 28 at 20:27
  • $\begingroup$ "Orthogonal" and "uncorrelated to a variable" are synonymous when an intercept is included in the regression. $\endgroup$ – whuber Jan 28 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.