9
$\begingroup$

A friend of mine recently asked what is so ordinary, about ordinary least squares. We did not seem to get anywhere in the discussion. We both agreed that OLS is special case of the linear model, it has many uses, is well know, and is a special case of many other models. But is this really all?

Therefore I would like to know:

  • Where did the name really come from?
  • Who was the first to use the name?
$\endgroup$
  • 5
    $\begingroup$ It is ordinary because there are other variants now. Weighted. Robust. nonlinear. ... Ordinary is how you keep it from being confused with something else. $\endgroup$ – EngrStudent May 8 '16 at 12:22
  • 1
    $\begingroup$ As a guess OLS is likely the first numerical fit algorithm developed circa 1805 AD. The name ordinary was likely added after variations on the theme were developed. $\endgroup$ – Carl Nov 18 '16 at 23:29
2
$\begingroup$

Least squares in $y$ is often called ordinary least squares (OLS) because it was the first ever statistical procedure to be developed circa 1800, see history. It is equivalent to minimizing the $L_2$ norm, $||Y-f(X)||_2$. Subsequently, weighted least squares, minimization of other norms (e.g., $L_1$), generalized least squares, M Estimation , bivariate minimization (e.g., Deming regression), non-parametric regression, maximum likelihood regression, regularization (e.g., Tikhonov, ridge) and other inverse problem techniques and multiple other tools were developed. There is still controversy over who first applied it, Gauss or Legendre (see link). The term "ordinary" (implying in $y$) was obviously added to "least squares" only after so many alternative methods arose that the (still most) popular OLS needed to be differentiated from the plethora of other minimizations that became available. When exactly adding ordinary$+$least squares occurred would be hard to track down since that occurred when it became natural or obvious to do so.

$\endgroup$
  • 1
    $\begingroup$ You can also contrast OLS with WLS, weighted least squares, which minimizes the weighted squares. $\endgroup$ – AdamO Dec 12 '16 at 21:33
  • $\begingroup$ Its actually in there somewhat already under "other" norms, for example, $||1-\frac{Y}{f(x)}||$ is weighted. $\endgroup$ – Carl Dec 12 '16 at 21:45
  • $\begingroup$ This is an old question, if I remember correctly I was really interested (at the time) to figure out who first used the term OLS... Guess I will never know... Good answer tho +1 $\endgroup$ – Repmat Dec 13 '16 at 9:36
  • $\begingroup$ I have never heard "least squares" used to describe minimization that is not an $L_2$ norm. (Note that "energy norms" such as used in Tikhonov regularization are just an $L_2$ of a transformed vector.) A more general term would be just "regression", or perhaps "M estimation". $\endgroup$ – GeoMatt22 Dec 17 '16 at 6:11
  • $\begingroup$ I agree with @AdamO that ordinary vs. weighted* seems the likely contrast. (*Rather, "Generalized". In computational science I think this would still just be called weighted? Why would a weight matrix have to be diagonal?) $\endgroup$ – GeoMatt22 Dec 17 '16 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.