Why is OLS related to Moment Estimation?

Question

I am trying to understand the relationship between OLS and the Method of Moments. Specifically, why OLS is considered a special case of Method of Moments ... and why Method of Moments is a special case of OLS.

I read that for a simple regression model:

$$y_i = \beta_0 + \beta_1x_i + \epsilon_i$$

To do moment estimation, we have assumptions about expected values of errors are 0 and errors are not correlated with x-variables:

$$E(\epsilon_i) = 0$$ $$E(x_i\epsilon_i) = 0$$

If we make substitutions, we get:

1)$$E(y_i - \beta_0 - \beta_1x_i) = 0$$ 2)$$E(x_i(y_i - \beta_0 - \beta_1x_i)) = 0$$

They say that the above equation is like OLS now because 1) looks like the Error term is being minimized in OLS. But the error is not squared. And second moment is supposed to be about variance but I do not see it in 2).

I don't fully understand the relationship between OLS and Method of Moments. What am I missing?

Answer 1

You get the equations you quote by differentiating the squared error.
If $$\mathrm{RSS}=\sum_i (y-\beta_0-\beta_1x)^2$$ then $$\frac{\partial\mathrm{RSS}}{\partial\beta_0} = -2\sum_i (y-\beta_0-\beta_1x)$$ and $$\frac{\partial\mathrm{RSS}}{\partial\beta_1} = -2\sum_i x_i(y-\beta_0-\beta_1x)$$

To minimise RSS, you set these derivatives to zero and get exactly your moment equations.

They are moment equations because they only depend on moments of $Y|X$ and don't depend on other aspects of the distribution.

You don't need to use variances when estimating $\beta$, so they doesn't show up in your moment equations. You would need to use the variances to estimate the residual variance, and then you'd have a third equation estimating $\sigma^2$ in terms of the variance of the residuals.

Answer 2

The idea of method of moments says that you

• take the population moment conditions (here, after substitution, the last two equations you present)
• replace expected values with sample analogons (sample means)
• and solve the (system of) equation(s) for the unknown parameter(s)

Here, we hence exploit the two sample moment conditions \begin{equation} \frac{1}{n}\sum_{i=1}^n (y_i-\tilde\beta_1-\tilde\beta_2x_i)=0 \end{equation} and \begin{equation} \frac{1}{n}\sum_{i=1}^n\bigl(x_i(y_i-\tilde\beta_1-\tilde\beta_2x_i)\bigr)=0. \end{equation} Rewriting yields \begin{eqnarray*} \tilde\beta_1+\left(\frac{1}{n}\sum_{i=1}^n x_i\right)\tilde\beta_2&=&\frac{1}{n}\sum_{i=1}^n y_i\\ \left(\frac{1}{n}\sum_{i=1}^n x_i\right)\tilde\beta_1+\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right)\tilde\beta_2&=&\frac{1}{n}\sum_{i=1}^n x_iy_i\\ \end{eqnarray*} Multiplying through with $n$ gives \begin{eqnarray*} n\tilde\beta_1+\left(\sum_{i=1}^n x_i\right)\tilde\beta_2&=&\sum_{i=1}^n y_i\\ \left(\sum_{i=1}^n x_i\right)\tilde\beta_1+\left(\sum_{i=1}^n x_i^2\right)\tilde\beta_2&=&\sum_{i=1}^n x_iy_i\\ \end{eqnarray*} In matrix notation, $$ \begin{pmatrix} n&\sum_{i=1}^n x_i\\ \sum_{i=1}^n x_i&\sum_{i=1}^n x_i^2 \end{pmatrix}\begin{pmatrix} \tilde\beta_1\\ \tilde\beta_2 \end{pmatrix}=\begin{pmatrix} \sum_{i=1}^n y_i\\ \sum_{i=1}^n x_iy_i \end{pmatrix} $$ or, more compactly, \begin{equation} X'X\tilde\beta=X'y \end{equation} Solving for $\tilde\beta$ gives the OLS estimator \begin{equation} \hat{\beta}_{ols}=\bigl(X'X\bigr)^{-1}X'y \end{equation} OLS is a spcial case of MoM as you can, e.g., infer from many other posts on this site that discuss applications of MoM.

Do you have a reference for why MoM would be a special case of OLS?