The Bias of Incorrectly Fit Model in a Linear Regression Model

Question

"Suppose that we have fit the straight-line regression model $\hat{y} = \hat{\beta_0} + \hat{\beta_1}x_1$ but the response is affected by a second variable $x_2$ such that the true regression function is

$$E(y) = \beta_0 + \beta_1x_1 + \beta_2x_2$$

Show the bias in $\hat{\beta}_1$

I worked as follows:

$$E(\hat{\beta_1})= E(\sum_{I=1}^n c_i y_i)$$

where

$$c_i = \frac{x_i-\bar{x}}{S_{xx}}$$

Since I know the true model I know that the expectation of $y_i$ is the model with two predictors. So when I expanded I got,

$$\sum_{I=1}^n c_i\beta_0 + \sum_{I=1}^n c_i \beta_1 x_{i1} + \sum_{I=1}^n \beta_2 c_i x_{i2}$$

The first term I know is zero since summing $c_i$ gets 0. But I also know the summing $c_ix_i$ equals 1. Does this mean that the expectation is $\beta_1 + \beta_2$?

Answer 1

You might want to have a look at some resources on omitted variable bias. Your question is the case that is commonly used in illustrations of this phenomenon. I will add the derivation of the relevant equations here, but you can get more information for broader cases if you read up on the general topic of omitted variable bias in regression.

---

Vector form for omitted variable bias: To find the bias, write your model in vector form as:

$$\boldsymbol{Y} = \boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{X}_2 \boldsymbol{\beta}_2 + \boldsymbol{\varepsilon},$$

where $\boldsymbol{X}$ is composed of a column of ones (for the intercept) and a column of values for $x_1$, and $\boldsymbol{X}_2$ is composed of a column of values for $x_2$. Now, take the OLS estimate for the first parameter (when we omit the second from the model), and substitute the true form of $\boldsymbol{Y}$ (which includes the omitted variable). This gives you the equation:

$$\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}} &= (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} \boldsymbol{Y} \\[6pt] &= (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} (\boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{X}_2 \boldsymbol{\beta}_2 + \boldsymbol{\varepsilon}) \\[6pt] &= \boldsymbol{\beta} + (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} (\boldsymbol{X}^\text{T} \boldsymbol{X}_2) \boldsymbol{\beta}_2 + (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} \boldsymbol{\varepsilon}. \\[6pt] \end{aligned} \end{equation}$$

Hence, the bias of the estimator is:

$$\begin{equation} \begin{aligned} \text{Bias}(\hat{\boldsymbol{\beta}} | \boldsymbol{X}, \boldsymbol{X}_2) &= \mathbb{E}(\hat{\boldsymbol{\beta}} | \boldsymbol{X}, \boldsymbol{X}_2) - \boldsymbol{\beta} \\[6pt] &= (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} (\boldsymbol{X}^\text{T} \boldsymbol{X}_2) \boldsymbol{\beta}_2 + (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{X}, \boldsymbol{X}_2) \\[6pt] &= (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} (\boldsymbol{X}^\text{T} \boldsymbol{X}_2) \boldsymbol{\beta}_2. \\[6pt] \end{aligned} \end{equation}$$

---

Scalar form: We can obtain scalar equations for the bias of each scalar estimator by substituting the design matrices into this equation and simplifying to scalar equations. We will endeavour to simplify this calculation to use standard terms for the sample means, variances, covariances, etc., and so it is worth noting that:

$$\begin{equation} \begin{aligned} s_1^2 &= \frac{\sum x_{1,i}^2}{n} - \bar{x}_1^2 & & & (\sum x_{1,i}^2) &= n (s_1^2 + \bar{x}_1^2), \\[10pt] s_2^2 &= \frac{\sum x_{2,i}^2}{n} - \bar{x}_2^2 & & & (\sum x_{2,i}^2) &= n (s_2^2 + \bar{x}_2^2), \\[10pt] r_{1,2} &= \frac{\sum x_{1,i} x_{2,i}}{n} - \bar{x}_1 \bar{x}_2 & & & (\sum x_{1,i} x_{2,i}) &= n (r_{1,2} + \bar{x}_1 \bar{x}_2). \\[10pt] \end{aligned} \end{equation}$$

Substituting the relevant design matrices and simplifying algebraically, we obtain:

$$\begin{equation} \begin{aligned} \text{Bias}(\hat{\boldsymbol{\beta}} | \boldsymbol{X}, \boldsymbol{X}_2) &= \begin{bmatrix} n & \sum x_{1,i} \\ \sum x_{1,i} & \sum x_{1,i}^2 \end{bmatrix}^{-1} \begin{bmatrix} \sum x_{2,i} \\ \sum x_{1,i} x_{2,i} \end{bmatrix} \beta_2 \\[6pt] &= \frac{1}{n \sum x_{1,i}^2 - (\sum x_{1,i})^2} \begin{bmatrix} \sum x_{1,i}^2 & -\sum x_{1,i} \\ -\sum x_{1,i} & n \end{bmatrix} \begin{bmatrix} \sum x_{2,i} \\ \sum x_{1,i} x_{2,i} \end{bmatrix} \beta_2 \\[6pt] &= \frac{1}{n \sum x_{1,i}^2 - (\sum x_{1,i})^2} \begin{bmatrix} (\sum x_{1,i}^2) (\sum x_{2,i}) - (\sum x_{1,i})(\sum x_{1,i} x_{2,i}) \\ n (\sum x_{1,i} x_{2,i}) - (\sum x_{1,i}) (\sum x_{2,i}) \end{bmatrix} \beta_2 \\[6pt] &= \frac{1}{\sum x_{1,i}^2 - n \bar{x}_1^2} \begin{bmatrix} (\sum x_{1,i}^2) \bar{x}_2 - \bar{x}_1 (\sum x_{1,i} x_{2,i}) \\ \sum x_{1,i} x_{2,i} - n \bar{x}_1 \bar{x}_2 \end{bmatrix} \beta_2 \\[6pt] &= \frac{1}{n s_1^2} \begin{bmatrix} (n s_1^2 + \bar{x}_1^2) \bar{x}_2 - \bar{x}_1 (n r_{1,2} + \bar{x}_1 \bar{x}_2) \\ n r_{1,2} \end{bmatrix} \beta_2 \\[6pt] &= \frac{1}{s_1^2} \begin{bmatrix} s_1^2 \bar{x}_2 - \bar{x}_1 r_{1,2} \\ r_{1,2} \end{bmatrix} \beta_2 \\[6pt] &= \begin{bmatrix} \bar{x}_2 - \bar{x}_1 \cdot \rho_{1,2} \cdot s_2/s_1 \\ \rho_{1,2} \cdot s_2/s_1 \end{bmatrix} \beta_2. \\[6pt] \end{aligned} \end{equation}$$

So we have:

$$\begin{equation} \begin{aligned} \text{Bias}(\hat{\beta}_0 | \boldsymbol{X}, \boldsymbol{X}_2) &= \Big( \bar{x}_2 - \bar{x}_1 \cdot \rho_{1,2} \cdot \frac{s_2}{s_1} \Big) \cdot \beta_2, \\[6pt] \text{Bias}(\hat{\beta}_1 | \boldsymbol{X}, \boldsymbol{X}_2) &= \rho_{1,2} \cdot \frac{s_2}{s_1} \cdot \beta_2. \\[6pt] \end{aligned} \end{equation}$$

As you can see from these equations, the bias in estimating the slope parameter $\beta_1$ comes from the correlation between the included and omitted explanatory variables, and the bias for estimating the intercept parameter $\beta_0$ comes from the correlation between these variables, plus the sample mean of the omitted variable. You will also notice that in the special case where the included and omitted explanatory variables are uncorrelated, all of the bias is absorbed into the intercept term.