Tag Info

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Overview

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Formulation

Given a set of data $(x_1,y_1),...,(x_n,y_n)$ where $x_i \in \mathbb{R}^{p}$ and a vector of coefficients $\beta$, the least squares estimate is the solution to the equation:

$$\widehat{\beta}_{LS} = \underset{\beta} {\text{arg min}} \sum\limits_{i=1}^{n}(y_i - \sum\limits_{j=1}^{p}x_{i,j}\beta_{j})^2 = || {\bf y - X\beta}||^2$$

Using linear algebra, one can find the least squares hyperplane:

$${\bf \widehat{\beta} = (X^TX)^{-1}X^{T}y}$$

References

Least squares methods are treated in many introductory statistics resources and textbooks, but there are also advanced resources dedicated only to the subject, for example: