11
$\begingroup$

Basic setup:

regression model: $y = \text{constant} +\beta_1x_1+\beta_2x_2+\beta_3x_3+\beta_4x_4+\alpha C+\epsilon$ where C is the vector of control variables.

I'm interested in $\beta$ and expect $\beta_1$ and $\beta_2$ to be negative. However, there is multicollinearity problem in the model, the coefficient of correlation is given by, corr($x_1$,$x_2)=$ 0.9345, corr($x_1$,$x_3)=$ 0.1765, corr($x_2$,$x_3)=$ 0.3019.

So $x_1$ and $x_2$ are highly correlated, and they should virtually provide the same information. I run three regressions:

  1. exclude $x_1$ variable; 2. exclude $x_2$ variable; 3. original model with both $x_1$ and $x_2$.

Results:
For regression 1 and 2, it provide the expected sign for $\beta_2$ and $\beta_1$ respectively and with similar magnitude. And $\beta_2$ and $\beta_1$ are significant in 10% level in both model after I do the HAC correction in standard error. $\beta_3$ is positive but not significant in both model.

But for 3, $\beta_1$ has the expected sign, but the sign for $\beta_2$ is positive with the magnitude twice greater than $\beta_1$ in absolute value. And both $\beta_1$ and $\beta_2$ are insignificant. Moreover, the magnitude for $\beta_3$ reduces almost in half compared to regression 1 and 2.

My question is:

Why in 3, the sign of $\beta_2$ becomes positive and much greater than $\beta_1$ in absolute value? Is there any statistical reason that $\beta_2$ can flip sign and has large magnitude? Or is it because model 1 and 2 suffer omitted variable problem which inflated $\beta_3$ provided $x_2$ has positive effect on y? But then in regression model 1 and 2, both $\beta_2$ and $\beta_1$ should be positive instead of negative, since the total effect of $x_1$ and $x_2$ in regression model 3 is positive.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
13
$\begingroup$

Think of this example:

Collect a dataset based on the coins in peoples pockets, the y variable/response is the total value of the coins, the variable x1 is the total number of coins and x2 is the number of coins that are not quarters (or whatever the largest value of the common coins are for the local).

It is easy to see that the regression with either x1 or x2 would give a positive slope, but when incuding both in the model the slope on x2 would go negative since increasing the number of smaller coins without increasing the total number of coins would mean replacing large coins with smaller ones and reducing the overall value (y).

The same thing can happen any time you have correlalted x variables, the signs can easily be opposite between when a term is by itself and in the presence of others.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Greg, how does this work with times series financial data? If I’m only interested in understanding the relationship between x1 and y, and x2 and y, then wouldn’t I opt for the separate regression interpretation? Certainly it is true that y tends to rise as x2 rises. Only in the context of holding x1 constant does x2 have a negative relationship with y. $\endgroup$
    – user3138766
    Sep 5, 2021 at 22:49
3
$\begingroup$

You have answered your own question - there is collinearity.

A bit of explanation: $x_1$ and $x_2$ are highly collinear. But when you enter both into the regression, the regression is attempting to control for the effect of the other variables. In other words, hold $x_1$ constant, what do changes in $x_2$ do to $y$. But the fact that they are so highly related means that this question is silly, and weird things can happen.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks a lot. But since multicolinearity in theory only inflatesthe variance but not affects the overall prediction power of the highly correlated variables, so I thought $\beta_1*x1+\beta_2*x2$ in model 3 should provide similar result as $\beta_2*x2$ in model 1 or $\beta_1*x1$ in model 2, since pairwise correlation of x1 x2 with x3 is not high (actually this is my confusing part). But since correlation can be really messy, and in practice, I should not expect this since my model is only an approximation of the DGP and the correlation with other variables matters. $\endgroup$
    – ting
    Oct 31, 2012 at 23:21
  • $\begingroup$ If you want to get into the math of this, I highly recommend books by David Belsley. $\endgroup$
    – Peter Flom
    Oct 31, 2012 at 23:39
  • $\begingroup$ Great, Thank you so much!!! Just requested the books from the library :) $\endgroup$
    – ting
    Oct 31, 2012 at 23:48
2
$\begingroup$

Why in 3, the sign of β2 becomes positive and much greater than β1 in absolute value? Is there any statistical reason that β2 can flip sign and has large magnitude?

The simple answer is there is no deep reason.

The way to think about it is that when multicollineary approaches perfect, the specific values that you end up obtaining from the fitting become more and more dependent on smaller and smaller details of the data. If you were to sample the same amount of data from the same underlying distribution and then fit, you could obtain completely different fitted values.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.