In the paragraph shown here (from "Least Squares Estimation" by S.A. Geer) I don't understand how we convert the least square estimation problem into the equation 4.

Could someone please elaborate?

LSE problem

  • $\begingroup$ You have to find such $\hat{\beta}$ that minimizes the square error so equation 4 is obtained by taking derivative $$\frac{\partial}{\partial \beta}\Vert Y - X\beta \Vert^2 = 0$$ $\endgroup$ Commented Feb 19, 2017 at 8:43
  • $\begingroup$ I would be surprised if this question weren't answered in the passage immediately following the one shown. $\endgroup$
    – whuber
    Commented Feb 19, 2017 at 18:19

1 Answer 1


An informal derivation : your model is

$$\mathbf y = \mathbf X \beta +\mathbf u$$

premultiply by $\mathbf X'$ to get

$$\mathbf X'\mathbf y = \mathbf X'\mathbf X \beta +\mathbf X'\mathbf u$$

Now pre-multiply by $(\mathbf X'\mathbf X)^{-1}$ to get

$$(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf y = (\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf X \beta +(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u$$

The error term is unknown so ignore the last term, and simplify to get

$$(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf y = \hat \beta $$

and it is only estimated and not the exact $\beta$, because we have ignored the term $(\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf u$.

The formal treatment that validates the optimality of this approach under a certain criterion, is what a comment suggested.

  • $\begingroup$ Than you very much. $\endgroup$
    – wmac
    Commented Feb 20, 2017 at 3:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.