Linear regression changes in form when including other predictors

Question

let $y_{i}$ be the response variable, and let $x_{i}$ be a predictor in a linear regression model. Suppose that we have the general model $y_{i}=B_{0}-B_{1}x_{1}$ now suppose that we include another predictor $x_{2}$ in the model and get the model to be $y_{i}=C_{0}+C_{1}x_{1}+C_{2}x_{2}$. In this particular instance the sign of the coefficient for the predictor $x_{1}$ has changed in signs from negative to positive. Under what circumstance would this occur? wouldn't the marginal relationships remain the same for the predictors irrespective of how many predictors are included?

Answer 1

An overspecified model will still give unbiased coefficients, so with enough data, you should not see the sign change. In practice, where data is finite, this can happen for a few reasons, and the most common I can think of is if $x_1,x_2$ are very similar (you can think of approaching the multicollinearity issue), in which case you can observe, in finite samples, cases where one of the coefficients is positive and the other is negative in an attempt to, intuitively, 'cancel out'. Eventually, if you observe enough data, you won't run into any issues unless there is actual multicollinearity, but for for finite sums, you can always make $x_2$ similar-enough to $x_1$ to create a problem like you asked about.

Below is an illustration in R

set.seed(582)

n = 1000
x1 = rnorm(n)
x2 = x1 + rnorm(n)*.0001

y = x1 + rnorm(n)

> lm(y ~ x1)

Call:
lm(formula = y ~ x1)

Coefficients:
(Intercept)           x1  
   -0.01879      1.00850  

> lm(y~ x1+x2)

Call:
lm(formula = y ~ x1 + x2)

Coefficients:
(Intercept)           x1           x2  
   -0.01894    -32.84030     33.84874  

Answer 2

This happens very often when your "weaker" variable is correlated with your "stronger" variable and also correlated with the residual of the dependent when regressed with your "stronger" variable and the two relationships are opposite. Hence, if you regress the dependent with the "weaker" variable alone, you may even get a significant coefficient, due to omitted variable bias (i.e. due to the weaker variable's correlation with the stronger one which is not included). However, when you then include the "strong" variable, the omitted variable bias disappears and the variables get the right signs.

Thus, I do not agree that $x_1,x_2$ need to be anywhere near fully co-linear for this to happen. They have to be correlated though. Here is an example in python:

As you can see the correlation between $x_1,x_2$ is not dramatically high and still we see that if we only include $x_2$ in the regression, it gets a significantly positive coefficient, due to omitted variable bias. However, once we add $x_1$ in the model, both variables' coefficients converge to their true values.

I believe that if you combine my example with the one posted below, you see a healthy and an unhealthy situation where this can happen. If you notice on the example posted by the user below, the coefficients go crazy when both variables are included and even though it is not printed, it is certain that the $R^2$ will be unaffected by the inclusion of the 2nd variable, which is in contrast with my example. Hence, I think that you can check which of the two examples your problem resembles and decide accordingly on whether your $x_1$ is useful in informatively adjusting the the influence $x_2$, like $x_2$ does for $x_1$ in my case or whether you have an artifact caused by high levels of multicolinearity, as the example of the user below.