Suppose you run a regression:

$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \epsilon_i$

but you believe that:

$x_{i1} = f(x_{i2})$

will this cause any issues for your estimation and interpretation of the regression coefficients?

The following source (http://www.swlearning.com/pdfs/chapter/0324289782_3.PDF) on p. 84 states that in a simple regression setting your estimated beta coefficient “measures the sample relationship between $y$ and $x_1$ after $x_2$ has been partialled out” (removed). My concern, more generally, is that if you have two controls that pick up much of the same variation of $y$—and you choose to include both—then the general effect that they both try to identify upon $y$ may show up as insignificant (since it is—crudely put—split across the two), this while only including one of the two controls could have resulted in a significant identification of the general effect.

Am I wrong in thinking along these lines? I recognise that omitting a variable that is correlated with both $y$ and a control $x$ results in an omitted variable problem/bias. I am, however, not sure if the omitted variable problem is still a “bias inducing problem” when we believe that e.g. $x_{i1} = f(x_{i2})$ holds.

Any thoughts/comments would be much appreciated.

PS: I note that there are several questions on regression specification and general insignificance, but I have not found one that directly addressed my concerns.

  • $\begingroup$ I think you are talking about polynomial regression, look at the wikipedia page and many many posts here about polynomial and quadratic for a special common case. $\endgroup$ – Steve Sep 30 '14 at 18:35
  • $\begingroup$ Reading some more it appears that this is a classic problem of multicollinearity (see: en.wikipedia.org/wiki/Multicollinearity). $\endgroup$ – Seb Sep 30 '14 at 19:21
  • 1
    $\begingroup$ Sounds more like a two-stages least squares or structural equation model problem. $\endgroup$ – Corey Sparks Oct 2 '14 at 2:35

If $X_1$ is an exact affine function of $X_2$, $X_1 = \alpha_0+\alpha_1X_2$, then the "problem" is that your regressor matrix will be singular and non-invertible, and so OLS estimation will break down. But there is a way out, since in reality in such a case you have

$$y= \beta_0 + \beta_1(\alpha_0+\alpha_1x_2) + \beta_2x_2 + u$$

$$\Rightarrow y = \gamma_0 + \gamma_1x_2 + u,\;\; \gamma_0 = \beta_0+\beta_1\alpha_0,\;\; \gamma_1=\beta_1\alpha_1+\beta_2$$

In other words, in such a case you can run a simple regression with only $X_2$ present as regressor, and you will obtain estimates for the $\gamma$ coefficients.

If $f(X_2)$ is not a linear function, in general, no problem arises. In fact very often one sees $f(X_2) = X_2^2$, i.e a regression of the form

$$y= \beta_0 + \beta_1x_2^2 + \beta_2x_2 + u$$

Remember, whether the regression is "linear or not" has to do with whether it is linear in the unknown coefficients, not in the regressors.

  • $\begingroup$ Thanks Alecos. That is a nice intuitive explanation. As I mentioned in a comment above this seems to be the standard issue of multicollinearity (when f(.) is taken to be linear). However, programs like Stata (to my knowledge) would often give a warning (and drop the variable causing issues) when this happens--I assume due to the invertability issue you mentioned. However, this is not the case in my application. With that said, what is the practical solution here? Do I drop one of the $x's$ from the regression in your example? $\endgroup$ – Seb Sep 30 '14 at 19:31
  • 1
    $\begingroup$ When you say "this is not the case in my application", do you mean "in my case, the $f()$ function is not linear" or "$f()$ is linear, but the software does not give any warning"? And if you mean the second, then how exactly does your software react? $\endgroup$ – Alecos Papadopoulos Sep 30 '14 at 20:02
  • $\begingroup$ Sorry for not being clear. I mean that f() is not linear since that would have been picked up by the software. I read some more about it and it seems like the problem is a bit like double edged sword in that there is a tradeoff--if I remove a variable then I increase the risk of biased coefficient estimates, while if I keep it in my coefficient estimates will be precise but the standard errors will be large (possibly yielding statistical insignificance where it would otherwise yield significance).I guess my question is: what do people do in practice to resolve this issue? $\endgroup$ – Seb Oct 1 '14 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.