Mostly Harmless Econometrics, explanation of solution to the population least squares problem

Question

I'm trying to make sense of Mostly Harmless Econometrics's explanation of the solution to the population least squares problem, but I'm not following Angrist and Pischke's argument. Here's what they write (on pg. 35):

This section is concerned with the vector of population regression coefficients, defined as the solution to a population least squares problem. At this point we are not worried about causality. Rather, we let the Kx1 regression coefficient vector $\beta$ be defined by solving $\beta = argmin_{b} E[(Y_{i} - X'_{i}b)^2]$. Using the first-order condition, $E[X_i(Y_i - X_i'b)] = 0$, the solution can be written $\beta = E(X_iX'_i)^{-1}E(X_iY_i)$. Note that by construction, $E(X_i(Y_i - X'_i\beta)) = 0$. In other words, the population residual, which we defined as $Y_i - X'_i\beta = e_i$, is uncorrelated with the regressors, $X_i$. It bears emphasizing that this error term does not have a life of its own. It owes its existence and meaning to $\beta$.

What I don't get is the sentence that starts "using the first-order condition." Why are we allowed to assume that first-order condition holds here?

I should mention that at this point in the book, Angrist and Pischke have already proved that a random variable $Y_i$ may be decomposed as $Y_i = E(Y_i | X_i) + \epsilon_i$, where $\epsilon_i$ is uncorrelated with any function of $X_i$. But I don't see how they can apply that theorem here.

Am I missing something simple, or is the book's explanation just confusing?

Answer 1

"First-order condition" in optimization just means taking the first derivative and setting it to zero. The second derivative here is clearly negative, so under mild regularity conditions the first-order condition will give you the optimal solution. You don't need to "assume" that equation--it's just something you want to solve that you know will give you an optimum under very general conditions. The population linear regression problem is by definition the solution to that equation.

Taking the first-order equation to be true directly implies that the residuals aren't correlated with $X$. You know the residuals have expectation zero, so $Cov(X,\epsilon) = E[X\epsilon] - E[X] E[\epsilon] = E[X\epsilon] = 0$ when you are solving the first-order condition.

I honestly agree with you that Mostly Harmless is confusing here and elsewhere. I'm a statistician, though, so I often find economists' ways of presenting stats frustrating ;P

Answer 2

It seems me that the logic of Authors explanation is the following.

I should mention that at this point in the book, Angrist and Pischke have already proved that a random variable $Y_i$ may be decomposed as $Y_i = E(Y_i | X_i) + \epsilon_i$, where $\epsilon_i$ is uncorrelated with any function of $X_i$. But I don't see how they can apply that theorem here.

The decomposition you mention is true regardless the OLS estimation procedure. The OLS are the best way for estimate parameters of the CEF under certain conditions, main important for this explanation: the true CEF is linear or we are focused on his linear approximation (read here for more details: Regression and the CEF). Then, OLS estimation procedure is summarized.

What I don't get is the sentence that starts "using the first-order condition." Why are we allowed to assume that first-order condition holds here?

We do not assume his validity, we impose his validity in order to solve the minimization problem. Precisely this imposition have the noted consequences about properties of residuals

Answer 3

The goal here is to show the usual OLS solution as arising from the method of moments (so a special case of a generalized method of moments estimator that is popular in economics).

The point is that you expect a certain condition to hold that has economic meaning, in this case, that each feature $x_i$ is uncorrelated with the error term. If that is the case, then $\mathbb E\left[x_i\varepsilon\right] = 0$ for each $x_i$. Method of moments estimation operates on the idea that, if we want a certain moment property to be true in the population, then we should expect it to be true on average in the sample, so we replace the expectation with the sample mean and the true error $\varepsilon$ with the empirical residuals that estimate the errors. Then we get that the empirical moment equals zero.

You don’t have to derive the OLS estimator this way. Indeed, it is fine to derive the OLS estimator as maximum likelihood estimation for a Gaussian likelihood, even just saying that you want to minimize the sum of squared residuals because you want to impose a more severe penalty for bad misses than absolute residual minimization gives, free of reference to likelihood or even moment conditions (in some sense, purely mathematical, not statistical). I find it elegant that all of these coincide.