2
$\begingroup$

I have a multiple regression problem. Let's say there is a physical system with a true model:

$$ y = b_0x_0 + b_1x_1 + b_2x_2 \;\;\;\;\;\;\;\;\;\; (1) $$

Now, imagine I only have access to a subset of the true independent variables, such that I fit a model (let's assume the modeling process is fully accurate to the given data) as:

$$ y' = b_0'x_0 + b_1'x_1 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; (2) $$

The model produces a set of non-zero errors because my observations do not contain values for $x_2$.

My primary goal is to understand the regression coefficients. My understanding is that $b_0'$ and $b_1'$ will be inflated compared to $b_0$ and $b_1$ (assuming all coefficients are positive) because the contribution of $x_2$ to the prediction will be partially incorporated into the contributions from $x_0$ and $x_1$ (though imperfectly, resulting in the model errors). This should happen even if all variables are uncorrelated.

I want to report that the variable $x_0$ contributes $b_0'x_0$ to $y$, but as we can see between (1) and (2), it seems like $b_0'$ is entirely dependent on the other selected independent variables. This means that I can increase it's contribution almost-arbitrarily simply by removing more independent variables from the modeling.

My other concern is this: if I only have model (2), how do I know if important $x_2$ or $x_3$ or $x_4$ terms (and so on) exist in model (1)? Do I just assume that I have a 'correct' model with sufficient independent variables when its error approximates 0? Furthermore, if I can never approximate 0 error, is there a technique that can calculate error of the coefficients (i.e. how the coefficients might decrease if we did have knowledge of these 'missing' variables that would produce 0 error)?.

This is probably a large topic, so please let me know if there is a particular name for these concepts that I could investigate further.

$\endgroup$
  • $\begingroup$ I am assuming for simplicity that all coefficients and variables are positive. So, if b2x2 has a positive contribution to y, then when x2 is omitted and the variables are uncorrelated, the effect should be a b2x2 increase of y relative to x0 and x1 in the model, meaning the coefficients must go up to compensate. I guess I am also assuming that there is no intercept term that would 'receive' this increase. $\endgroup$ – ricky116 Aug 10 '18 at 13:28
1
$\begingroup$

The name for this phenomenon is omitted variable bias, with over 300 posts on this site.

Your intuition is close to correct but not completely accurate. I find this answer by @Gung to be a good point of entry into discussion about the problem. Strictly, if an omitted predictor variable* is uncorrelated with the other predictors in standard linear regression, then there should be no bias in the estimates of the coefficients of the others. In practice, there is almost always some correlation among predictors. Also, there may be bias in the intercept if the sample at hand was biased in terms of its inclusion of values of the omitted predictor.

This becomes even more of a problem with logistic regression or survival analysis with Cox regression. In these cases omitted variable bias is essentially unavoidable, even if predictors are uncorrelated. This page is a good place to start.


*As discussed on this page the term "independent variable" can be misleading in many contexts.

$\endgroup$
  • $\begingroup$ This is great, thanks! I tried many variations of different terms, 'omitted' was one I didn't think of. I'll look into this and your links. $\endgroup$ – ricky116 Aug 10 '18 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.