# How to derive OLS estimator for binary or bivariate regressor in terms of population moments

I am wondering how to find an OLS estimator for beta_1 in terms of population moments when the corresponding independent variable is a Bernoulli random variable equal to either 0 or 1. My professor gave the following formula during class (shown below). Any guidance would be very appreciated!

This is what I have so far, working from the formula provided:

Alternatively:

• Drawing a scatterplot will make very short work of this exercise, because it will make immediate the insight that the fitted line must pass through the two mean values.
– whuber
Jul 15, 2020 at 20:22
• See stats.stackexchange.com/questions/439092/… for an answer.
– whuber
Jul 19, 2020 at 20:42

## 1 Answer

Since this is clearly a homework problem, I won't give away the full answer. First, I'll note that the formula you posted does not appear to be correct; the denominator should be $$\sum(D_i-\bar D)^2$$. $$\sum(D_i-\bar D)$$ is just $$0$$.

The formula says that a sample estimate of $$\beta_1$$, namely, $$\hat \beta_1$$, is equal to the true value of $$\beta_1$$ in the population plus a bias term. The bias term is a ratio of two quantities, which involve $$D_i$$, the binary predictor; its mean $$\bar D$$ (also equal to the proportion of $$1$$s; and $$\epsilon_i$$, a random disturbance. The bias term can be written in terms of population moments (e.g., mean, variance, covariance, skew, etc.), and that is what you are being asked to figure out how to do.

I'll provide two insights that might help you:

1. $$\epsilon_i$$ is centered at $$0$$, meaning $$\epsilon_i = \epsilon_i - \bar \epsilon$$.
2. You can multiply the numerator and denominator by $$\frac{1}{n}$$, that is, $$\frac{1/n}{1/n} = 1$$, which seems trivial but is actually critical in converting the numerator and denominator into values that resemble population parameters.

Try using these clues to solve the problem. Let me know if you need more hints.

The original formula you presented was $$\hat\beta_1=\beta_1 + \frac{\sum(D_i - \bar D)\epsilon_i}{\sum (D_i - \bar D)^2}$$

When you replace $$\epsilon_i$$ with $$\epsilon_i - \bar \epsilon$$, the numerator is $$\sum(D_i - \bar D)(\epsilon_i - \bar \epsilon)$$. Hopefully that looks like a familiar formula, though, again, you need the second insight to make the connection. Using the second insight, you can write $$\frac{\sum W}{\sum Z}$$ as $$\frac{\frac{1}{n}\sum W}{\frac{1}{n}\sum Z}$$ (here I simply used $$W$$ and $$Z$$ as placeholders). So you can turn a ratio of sums into a ratio of means. What are they the means of in this case?

Here's what you are supposed to see:

$$\sum(D_i - \bar D)\epsilon_i = \sum(D_i - \bar D)(\epsilon_i - \bar \epsilon)$$ \begin{aligned} \hat\beta_1 &= \beta_1 + \frac{\sum(D_i - \bar D)\epsilon_i}{\sum (D_i - \bar D)^2} \\ &= \beta_1 + \frac{\sum(D_i - \bar D)(\epsilon_i - \bar \epsilon)}{\sum (D_i - \bar D)^2} \\ &= \beta_1 + \frac{\frac{1}{n}\sum(D_i - \bar D)(\epsilon_i - \bar \epsilon)}{\frac{1}{n}\sum (D_i - \bar D)^2} \\ &= \beta_1 + \frac{\text{Cov}(D, \epsilon)}{\text{Var}(D)} \end{aligned}

That's it. $$\text{Cov}(D, \epsilon)$$ and $$\text{Var}(D)$$ are moments of the joint distribution of $$D$$ and $$\epsilon$$. The usual OLS estimator for univariable regression is $$\frac{\text{Cov}(D, Y)}{\text{Var}(D)}$$; expanding $$Y$$ into $$Y = \beta_0 + \beta_1 D + \epsilon$$ will reduce to the formula I derived above.

Expanding the equation for $$Y$$ into the usual OLS solution:

\begin{aligned} \hat\beta_1 &= \frac{\text{Cov}(D, Y)}{\text{Var}(D)} \\ & = \frac{\text{Cov}(D, \beta_0 + \beta_1 D + \epsilon)}{\text{Var}(D)} \\ & = \frac{\text{Cov}(D, \beta_0) + \text{Cov}(D, \beta_1 D) + \text{Cov}(D, \epsilon)}{\text{Var}(D)} \\ &= \frac{0 + \beta_1 \text{Cov}(D, D) + \text{Cov}(D, \epsilon)}{\text{Var}(D)} \\ &= \frac{\beta_1 \text{Var}(D)}{\text{Var}(D)} + \frac{\text{Cov}(D, \epsilon)}{\text{Var}(D)} \\ &= \beta_1 + \frac{\text{Cov}(D, \epsilon)}{\text{Var}(D)} \end{aligned}

• Thank you for the response. I updated my question to show what I have done thus far. I'm unsure how those two insights factor in, as I end up finding that the bias term cancels out, leaving me only with the true value of 𝛽1. I'm not sure if this is what was supposed to happen, though, and I'm not sure if it's in terms of population moments. It's in terms of expected value, so it is in terms of mean; I assume, then, that it is in terms of population moments? Jul 15, 2020 at 18:06
• I made some edits. I think your math is a little incorrect, but trying going down the route I presented. In this case, simplification comes not from expanding terms into their parts but rather from recognizing a complicated formula represents a simple nameable quantity.
– Noah
Jul 15, 2020 at 18:40
• I'm not sure what multiplying the numerator and denominator by 1/n does. I could pull out the terms inside the summation so that I have 1/n multiplied by n multiplied by the mean of all terms inside the summation, but I don't understand what this is giving me. The formula does look similar to the expected value of beta_1 in the univariate case. Why can't I just use this formula except with D instead of X and using Y as usual? Jul 16, 2020 at 4:34
• See my last edit.
– Noah
Jul 16, 2020 at 5:11
• Wouldn’t solving this problem by expanding Y result in not having to include the beta_1 term? And isn’t it an issue that the beta_ term is still present? Or should I just write the beta_1 term in terms of expected values? Jul 16, 2020 at 15:45