16
$\begingroup$

Say I have a multivariable (several independent variables) regression that consists of 3 variables. Each of those variables has a given coefficient. If I decide to introduce a 4th variable and rerun the regression, will the coefficients of the 3 original variables change?

More broadly: in a multivariable (multiple independent variables) regression, is the coefficient of a given variable influenced by the coefficient of another variable?

$\endgroup$
  • 1
    $\begingroup$ Please edit the question to be more precise. Do by multivariable you mean multiple independent variables ("multiple regression") or multiple dependent variables ("multivariate regression" or "MAN(C)OVA")? $\endgroup$ – ttnphns Mar 13 '13 at 9:25
  • 1
    $\begingroup$ If the answer were no, there would not be a need to do multivariable regression in the first place! (we could simply do many univariable ones) $\endgroup$ – user603 Mar 13 '13 at 16:24
  • 1
    $\begingroup$ That's an insightful point, @user603, but I think there might still be a place for multiple regression, in that if the other variables were meaningfully related to the response (albeit not the explanatory variable), they can reduce the residual variance leading to improved power & precision. $\endgroup$ – gung - Reinstate Monica Mar 13 '13 at 16:30
23
$\begingroup$

A parameter estimate in a regression model (e.g., $\hat\beta_i$) will change if a variable, $X_j$, is added to the model that is:

  1. correlated with that parameter's corresponding variable, $X_i$ (which was already in the model), and
  2. correlated with the response variable, $Y$

An estimated beta will not change when a new variable is added, if either of the above are uncorrelated. Note that whether they are uncorrelated in the population (i.e., $\rho_{(X_i, X_j)}=0$, or $\rho_{(X_j, Y)}=0$) is irrelevant. What matters is that both sample correlations are exactly $0$. This will essentially never be the case in practice unless you are working with experimental data where the variables were manipulated such that they are uncorrelated by design.

Note also that the amount the parameters change may not be terribly meaningful (that depends, at least in part, on your theory). Moreover, the amount they can change is a function of the magnitudes of the two correlations above.

On a different note, it is not really correct to think of this phenomenon as "the coefficient of a given variable [being] influenced by the coefficient of another variable". It isn't the betas that are influencing each other. This phenomenon is a natural result of the algorithm that statistical software uses to estimate the slope parameters. Imagine a situation where $Y$ is caused by both $X_i$ and $X_j$, which in turn are correlated with each other. If only $X_i$ is in the model, some of the variation in $Y$ that is due to $X_j$ will be inappropriately attributed to $X_i$. This means that the value of $X_i$ is biased; this is called the omitted variable bias.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Very good point to make in that last sentence. $\endgroup$ – Glen_b Mar 13 '13 at 22:09
  • $\begingroup$ I discuss the flip side of this issue in my answer here: Estimating $b_1x_1+b_2x_2$ instead of $b_1x_1+b_2x_2+b_3x_3$. $\endgroup$ – gung - Reinstate Monica Sep 18 '14 at 0:25
  • $\begingroup$ @gung i know your answer is old but i just tried this ideone.com/6CAkSR where i created $y$ and $x2$ are correlated and $x1$ is uncorrelated with $y$. But when i added $x1$ to the model, the parameter of x2 changed although $x1$ is uncorrelated with $y$. you said in your answer "correlated with the response variable, $Y$ An estimated beta will not change when a new variable is added, if either of the above are uncorrelated.". Am i wrong? $\endgroup$ – floyd May 9 '18 at 20:05
  • 1
    $\begingroup$ It needs to be perfectly uncorrelated, not just not significantly correlated, @floyd. If so, the beta for $s_1$ should not have changed unless there was some error. $\endgroup$ – gung - Reinstate Monica May 10 '18 at 0:34
  • $\begingroup$ @gung thanks so much for replying back. Do you know a way of creating such perfect data? i know that can't happen in real life $\endgroup$ – floyd May 10 '18 at 9:13
3
$\begingroup$

It is mathematically possible that the coefficients will not change, but it is unlikely that there will be no change at all with real data, even if all the independent variables are independent of each other. But, when this is the case, the changes (other than in the intercept) will tend to 0:

set.seed(129231)
x1 <- rnorm(100)
x2 <- rnorm(100)
x3 <- rnorm(100)
x4 <- rnorm(100)
y <- x1 + x2 + x3 + x4 + rnorm(100, 0, .2)
lm1 <- lm(y~x1+x2+x3)
coef(lm1)
lm2 <- lm(y~x1+x2+x3+x4)
coef(lm2)

In the real world, though, independent variables are often related to each other. In this case, adding a 4th variable to the equation will change the other coefficients, sometimes by a lot.

Then there are possible interactions.... but that's another question.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Generally speaking, yes, adding a variable changes the earlier coefficients, almost always.

Indeed, this is essentially the cause of Simpson's paradox, where coefficients can change, even reverse sign, because of omitted covariates.

For that not to happen, we'd need that the new variables were orthogonal to the previous ones. This often happens in designed experiments, but is very unlikely to happen in data where the pattern of the independent variables is unplanned.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.