# Method of moments for linear regression?

I have been reading about the method of moments, and now I understand how to obtain the method of moments estimator for a random sample $$x_1,...,x_n$$ from a distribution $$f(x;\theta)$$, in the multiparameter case. However, I fail to see how to formulate the method of moments for the linear regression model $$y_i = x_i^T\beta + e_i,$$ where $$E[e_i]=0$$, $$\beta,xi\in R^p$$. There is this moment condition, but I cannot link it to a system of equations as in least squares estimation.

Is there a reference with this formulation?

• To derive a moment condition, you can use the first order condition from the least squares optimization problem, or the orthogonal projection theorem. – Frank Nov 9 '18 at 14:22

## 2 Answers

The least squares estimator is the solution to the estimating equation:

$$0 = \mathbf{X}^T \left( Y - \mathbf{X}\beta \right)$$

Where $$\mathbf{X} = [1, x_1, x_2, \ldots, x_p]$$ is a $$n \times p$$ model matrix of covariate(s).

This is a trivial result, but a more general discussion on estimating equations can be found in Wakefield "Bayesian and Frequentist Regression Methods". Estimating Equations are also called M-estimators. Another reference would be Boos, Stefanski "Essential Statistical Inference" ch. 7.

• I would add the qualitative comment that this expresses/enforces a condition of orthogonality between (population) residuals and explanatory variables which says "there is nothing omitted from the model that is correlated to something that has been included". – user8948 Nov 12 '18 at 13:56

@AdamO indicates the solution using OLS, which you seem to know already. In order to obtain the desired formulation, you need to use the Generalised Method of Moments, in order to facilitate this formulation.

https://en.wikipedia.org/wiki/Generalized_method_of_moments#Scope

The idea is to start with the equation

$$\operatorname {E} [\,x_{t}(y_{t}-x_{t}^{\mathsf {T}}\beta )\,]=0}$$

and produce the equivalent sample equation.