# Omitted Variables and their consequences for other variables

Suppose we have the following regression model: $$y_{i}=\boldsymbol{x_{i}'\beta}+e_{i}$$ where the vector $$\boldsymbol{x_{i}'}$$ contains two variables, $$[x_{1i}\,x_{2i}].$$Suppose for a second, that $$e_{i}$$ contains variables that correlated with $$x_{1i}$$. The well known omitted variables bias tells us that $$\beta_{1}$$ will be measured with bias when estimated by OLS. However, what implications does this have for $$\beta_{2}?$$ In other words, does bias in one set of estimates still have implications for bias in a second set of estimates? In other words, if $$cov\left(x_{1i},e_{i}\right)\neq0$$ , but $$cov\left(x_{2i},e_{i}\right)=0,$$ what does this mean for the estimates of $$\beta_{2}?$$

• Any updates? Seems like a straightforward question... – ChinG Nov 24 '19 at 15:50
• I think OLS will work fine for the coefficient $\beta_2$. No bias there. – user2338823 Nov 25 '19 at 16:31

Intuitively, the answer is that $$\hat{\beta}_2$$ will be biased whenever $$x_2$$ is correlated with $$x_1$$. This is because the correlation of $$x_2$$ with $$x_1$$ causes $$\hat{\beta}_2$$ to be correlated with $$\hat{\beta}_1$$; as a consequence, when $$\hat{\beta}_1$$ differs from $$\beta_1$$, $$\hat{\beta}_2$$ will differ from $$\beta_2$$ in expectation, and, writing loosely, since we expect $$\hat{\beta}_1$$ to be biased, $$\hat{\beta}_2$$ will be too.

I'll assume away the intercept term for simplicity of exposition, without loss of generality. Writing out the expression for $$\mathbb{E}\hat{\beta}$$ gives us:

$$\mathbb{E}\hat{\beta} = \mathbb{E}(X^TX)^{-1}X^TY = \mathbb{E}(X^TX)^{-1}X^T(X\beta + e) = \beta + \mathbb{E}(X^TX)^{-1}X^Te$$

Under the assumptions of the question, $$\mathbb{E}X^Te$$ can be written as $$(\theta,0)$$, where $$\theta$$ is a function of the nonzero covariance between $$x_1$$ and $$e$$. Substituting gives us:

$$\mathbb{E}\hat{\beta} - \beta = [(X^TX)^{-1}_{1,1}\theta, (X^TX)^{-1}_{2,1}\theta]$$

The subscripts designate which element of the $$2 \times 2$$ matrix $$(X^TX)^{-1}$$ is being referred to.

Since $$(X^TX)^{-1}_{1,1} > 0$$ unless $$x_1$$ is identically equal to $$0$$, we can see that $$\hat{\beta}_1$$ will be biased; however, $$\hat{\beta}_2$$ will be biased if and only if $$(X^TX)^{-1}_{2,1} \ne 0$$, i.e., if $$x_1$$ and $$x_2$$ are correlated.

Here's a simple example in R, where the true values of $$\beta_1$$ and $$\beta_2$$ are both equal to $$1$$:

bhat_1 <- rep(0, 10000)

V <- matrix(c(1,0.6,0.6,1), 2, 2)

for (i in 1:length(bhat_1)) {
x <- rmvn(100, c(0,0), V)
y <- x[,1] + x[,2] + rnorm(100)
bhat_1[i] <- coef(lm(y~x[,1]))
}

hist(bhat_1)


that produces the following histogram: • Your intuition is correct, and i'll edit my answer to put a more intuitive gloss on what's going on. – jbowman Nov 25 '19 at 20:41
• @ChinG What seems to be landing with my intuition is that I can reorder the variables $x_{1}$ and $x_{2}$ in a multiple regression (i.e. so that I am not omitting variables) so that the data vectors swap names, and still expect to see identical $\beta$s and SEs. If that is the case, then if there is omitted variables bias for $\beta_{1}$ due to omission of $x_{2}$, I would guess that the the same must be true for $\beta_{2}$ if I omit $x_{1}$. Correlations are non-directional, after all. Of course, my intuition might have the utility of a stopped clock here... :) (Also: +1 jbowman) – Alexis Nov 25 '19 at 21:12
• @Alexis - your intuition is correct; it really comes down to the structure of $X^TX$, and reordering the variables just rearranges the rows and columns, it doesn't change the covariances themselves. – jbowman Nov 25 '19 at 21:17
• This exposition is quite damning. In all the economics seminars I frequent, people only care about whether or not the variable of interest is exogenous to unobserved factors (i.e. if the variable can be interpreted in a causal sense). According to the line of reasoning displayed by @jbowman , one has to make sure that all the included variables are conditionally exogenous, otherwise mild correlation between regressors can render the coefficients on each one as biased. – ChinG Nov 25 '19 at 22:10
• Sad but true, and not only biased but also inconsistent. – jbowman Nov 25 '19 at 22:15

Here's what happens for any number of predictors and omitted variables. The answer ends up similar to what @jbowman (+1) showed for two predictors (one correlated with omitted variables and one not). Since I'd already started writing this, I'll post it in case it's useful to see another approach.

To summarize the answer below: Suppose $$X_u$$ contains predictors that are uncorrelated with all omitted variables and $$X_c$$ contains predictors that are correlated with at least one omitted variable. Then, under standard assumptions, OLS coefficients for predictors in $$X_u$$ are unbiased if and only if $$X_u$$ and $$X_c$$ are uncorrelated.

### Setup

Let $$y \in \mathbb{R}^n$$ be a vector of responses and $$X \in \mathbb{R}^{n \times p}$$ be a matrix of predictors. We also have predictors $$Z \in \mathbb{R}^{n \times q}$$ that will be omitted. Assume the predictors and responses have been centered, so there's no need for an intercept term. I'll also assume that predictors may be correlated but not perfectly collinear (this implies that $$X$$ and $$Z$$ have full rank, and $$n \ge p+q$$).

Suppose true model is:

$$y = X w + Z v + \epsilon \quad \quad \epsilon \sim \mathcal{N}(\vec{0}, \sigma^2 I)$$

where $$w \in \mathbb{R}^p$$ and $$v \in \mathbb{R}^q$$ are the true coefficients, and $$\epsilon$$ is a random vector representing i.i.d. Gaussian noise with mean zero and variance $$\sigma^2$$.

Say we fit an ordinary least squares regression model, omitting $$Z$$. The estimated coefficients are:

$$\hat{w} = (X^T X)^{-1} X^T y$$

The bias is a vector containing the expected difference between the estimated and true coefficients (see derivation below):

$$\text{bias} = E[\hat{w} - w] = (X^T X)^{-1} X^T Z v$$

Multiplying both sides by $$X^T X$$ gives:

$$X^T X \ \text{bias} = X^T Z v$$

### For predictors uncorrelated with the omitted variables

Suppose the predictors are partitioned as $$X = [X_u, X_c]$$, where $$X_u$$ contains columns uncorrelated with all omitted variables and $$X_c$$ contains columns correlated with at least one omitted variable. So, $$X_u^T Z = \mathbf{0}$$ and $$X_c^T Z \ne \mathbf{0}$$. Similarly, suppose the bias is partitioned into subvectors $$\text{bias}_u$$ (for predictors in $$X_u$$) and $$\text{bias}_c$$ (for predictors in $$X_c$$). Rewrite the preceding equation in the partitioned form:

$$\begin{bmatrix} X_u^T X_u & X_u^T X_c \\ X_c^T X_u & X_c^T X_c \end{bmatrix} \begin{bmatrix} \text{bias}_u \\ \text{bias}_c \end{bmatrix} = \begin{bmatrix} \mathbf{0} \\ X_c^T Z \end{bmatrix} v$$

Break this into two systems:

$$X_u^T X_u \text{bias}_u + X_u^T X_c \text{bias}_c = \vec{0}$$

$$X_c^T X_u \text{bias}_u + X_c^T X_c \text{bias}_c = X_c^T Z v$$

$$\text{bias}_c$$ is nonzero, since assuming it's zero leads to a contradiction, given our assumptions. This recapitulates the standard statement about omitted variable bias.

More interestingly, the question concerns $$\text{bias}_u$$, the bias of predictors that are uncorrelated with the omitted variables. The first equation in the above pair leads to two conclusions: 1) If $$X_u$$ and $$X_c$$ are uncorrelated so $$X_u^T X_c = \mathbf{0}$$, then the only solution is $$\text{bias}_u = \vec{0}$$. Recall that $$X$$ has full rank so $$X_u$$ does too, and the null space of $$X_u^T X_u$$ includes only the zero vector. 2) Since $$\text{bias}_c$$ is nonzero, $$\text{bias}_u = \vec{0}$$ would imply that $$X_u^T X_c = \mathbf{0}$$.

Therefore, coefficients for predictors in $$X_u$$ have zero bias if and only if $$X_u$$ and $$X_c$$ are uncorrelated:

$$\text{bias}_u = \vec{0} \ \iff \ X_u^T X_c = \mathbf{0}$$

### Derivation of the bias

The bias is the expected difference between the estimated and true coefficients, where the expectation is taken w.r.t. $$\epsilon$$:

$$\begin{array}{ccl} \text{bias} & = & E[\hat{w} - w] \\ & = & E\big[ (X^T X)^{-1} X^T y - w \big] \\ & = & E\big[ (X^T X)^{-1} X^T X w + (X^T X)^{-1} X^T Z v + (X^T X)^{-1} X^T \epsilon - w \big] \\ & = & E\big[ (X^T X)^{-1} X^T Z v + (X^T X)^{-1} X^T \epsilon \big] \\ & = & (X^T X)^{-1} X^T Z v + E\big[ (X^T X)^{-1} X^T \epsilon \big] \\ & = & (X^T X)^{-1} X^T Z v \\ \end{array}$$

Line 2 substitutes in the expression for $$\hat{w}$$. Line 3 substitutes the true model in for $$y$$. Line 4 is an algebraic simplification. Line 5 uses linearity of expectation. Since noise has zero mean and is uncorrelated with the predictors, the expectation in the last term is zero, giving line 6.