Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Overview

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Formulation

Given a set of data $(x_1,y_1),...,(x_n,y_n)$ where $x_i \in \mathbb{R}^{p}$ and a vector of coefficients $\beta$, the least squares estimate is the solution to the equation:

$$\widehat{\beta}_{LS} = \underset{\beta} {\text{arg min}} \sum\limits_{i=1}^{n}(y_i - \sum\limits_{j=1}^{p}x_{i,j}\beta_{j})^2 = || {\bf y - X\beta}||^2$$

Using linear algebra, one can find the least squares hyperplane:

$$ {\bf \widehat{\beta} = (X^TX)^{-1}X^{T}y} $$

References

Least squares methods are treated in many introductory statistics resources and textbooks, but there are also advanced resources dedicated only to the subject, for example:

history | excerpt history