7
$\begingroup$

I see it is often quoted that the omitted variable bias formula is

$$ \text{Bias}\left(\widehat{\beta_1}\right) = \beta_2 \cdot \text{Corr}\left(X_2,X_1\right) $$

where $\widehat{\beta_1}$ is the estimated coefficient in the biased model, $\beta_2$ is the true coefficient of the omitted variable $X_2$ in the full model.

I am wondering how this is derived generally. Thanks.

$\endgroup$
2
  • 1
    $\begingroup$ A little units analysis suggests this formula is not correct, but it will be if we assume these are standardized coefficients. After all, the units of measurement of $\hat\beta_1$ and $\beta_2$ usually differ (the first is in units of the response per unit of $X_1$ while the second is units of response per unit of $X_2$) but a correlation is unitless. Thus, your equation can possibly hold only in circumstances where $X_1$ and $X_2$ have the same units, which would not be a general formula. $\endgroup$
    – whuber
    Apr 10, 2023 at 19:17
  • $\begingroup$ @whuber Thank you, that makes sense. $\endgroup$
    – user321627
    Apr 10, 2023 at 19:24

1 Answer 1

7
$\begingroup$

Here's an example of how one analyzes such a situation. Suppose that the model

$$E[Y\mid X_1,X_2] = \beta_1 X_1 + \beta_2 X_2$$

holds where $X_1,$ $X_2,$ and $Y$ are random $n$-vectors. If you omit the second variable and use the wrong model (omitting the $X_2$ variable)

$$E[Y\mid X] = \gamma_1 X_1,$$

we may ask what error you might expect in using an estimate of $\gamma_1$ to estimate $\beta_1.$ When you use ordinary least squares regression to estimate $\gamma_1,$ the formula is

$$\hat\gamma_1 = \frac{Y\cdot X_1}{X_1\cdot X_1}.$$

Using the correct model formulation you may compute

$$E[\hat\gamma_1\mid X_1, X_2] = E\left[\frac{Y\cdot X_1}{X_1\cdot X_1}\mid X_1,X_2\right] = E\left[\frac{(\beta_1 X_1 + \beta_2 X_2)\cdot X_1}{X_1\cdot X_1}\mid X_1,X_2\right].$$

Basic properties of expectation (linearity) and conditional expectation (taking out what is known) allow you to simplify the right hand side to

$$E[\hat\gamma_1\mid X_1, X_2] = \beta_1 + \beta_2 \frac{X_2\cdot X_1}{X_1\cdot X_1}.$$

By definition, the (conditional) bias in an estimate is the difference between its expectation and estimand,

$$\text{bias} = E[\hat\gamma_1\mid X_1,X_2] - \beta_1 = \beta_2 \frac{X_2\cdot X_1}{X_1\cdot X_1}.$$

That is a general formula, applicable for fixed $X_i$ or for random $X_i$ where $X_1\cdot X_1$ is almost surely nonzero. Already the result is helpful, because it implies that when $X_1$ is orthogonal to $X_2$ (which is just a way of saying the numerator is zero), the bias is zero; and otherwise it shows that the bias is nonzero and it gives you information about its sign and magnitude.

If, additionally, you arrange for the $X_i$ to be standardized (which means their components sum to zero and the squares of their components sum to unity), the fraction on the right could be called the "correlation" of $X_1$ and $X_2,$ understanding this term to be a shorthand for

$$\operatorname{correlation}(X_1,X_2) = \frac{\sum_{i=1}^n X_{1i}X_{2i}}{\sum_{i=1}^n X_{1i}X_{1i}} = \frac{\sum_{i=1}^n X_{1i}X_{2i}}{1} = \sum_{i=1}^n X_{1i}X_{2i},$$

which is the Pearson correlation of standardized vectors. But please note that this is not the correlation in the sense that $X_1$ and $X_2$ might be random vectors: the bias was computed conditionally and still depends on $X_1$ and $X_2.$ If, for instance, each $X_i$ were an iid sequence of random values drawn from a bivariate distribution, that underlying distribution can have a correlation but it's unlikely to equal the value computed in the bias formula.

$\endgroup$
1
  • $\begingroup$ Thank you so very much for this. $\endgroup$
    – user321627
    Apr 12, 2023 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.