# What is ordinary, in ordinary least squares?

A friend of mine recently asked what is so ordinary, about ordinary least squares. We did not seem to get anywhere in the discussion. We both agreed that OLS is special case of the linear model, it has many uses, is well know, and is a special case of many other models. But is this really all?

Therefore I would like to know:

• Where did the name really come from?
• Who was the first to use the name?
• It is ordinary because there are other variants now. Weighted. Robust. nonlinear. ... Ordinary is how you keep it from being confused with something else. May 8, 2016 at 12:22
• As a guess OLS is likely the first numerical fit algorithm developed circa 1805 AD. The name ordinary was likely added after variations on the theme were developed.
– Carl
Nov 18, 2016 at 23:29

## 1 Answer

Least squares in $$y$$ is often called ordinary least squares (OLS) because it was the first ever statistical procedure to be developed circa 1800, see history. It is equivalent to minimizing the $$L_2$$ norm, $$||Y-f(X)||_2$$. Subsequently, weighted least squares, minimization of other norms (e.g., $$L_1$$), generalized least squares, M Estimation, bivariate minimization (e.g., Deming regression), non-parametric regression, maximum likelihood regression, regularization (e.g., Tikhonov, ridge) and other inverse problem techniques and multiple other tools were developed. There is still controversy over who first applied it, Gauss or Legendre (see link). The term "ordinary" (implying in $$y$$) was obviously added to "least squares" only after so many alternative methods arose that the (still most) popular OLS needed to be differentiated from the plethora of other minimizations that became available. The word ordinary is often used in mathematical jargon as a synonym of simple. For example, consider the phrase ordinary differential equations. When exactly adding ordinary$$+$$least squares occurred would be hard to track down since that occurred when it became natural or obvious to do so. EDIT (for @alexis): Early 20th century I would presume, @GeoMatt22 says 1921 in comments below, but the mathematical meaning of ordinary (sive, vanilla) predates that and certain differential equations were called ordinary in the late 18th century as per The History of Differential Equations, 1670--1950.

• You can also contrast OLS with WLS, weighted least squares, which minimizes the weighted squares. Dec 12, 2016 at 21:33
• Its actually in there somewhat already under "other" norms, for example, $||1-\frac{Y}{f(x)}||$ is weighted.
– Carl
Dec 12, 2016 at 21:45
• This is an old question, if I remember correctly I was really interested (at the time) to figure out who first used the term OLS... Guess I will never know... Good answer tho +1 Dec 13, 2016 at 9:36
• I have never heard "least squares" used to describe minimization that is not an $L_2$ norm. (Note that "energy norms" such as used in Tikhonov regularization are just an $L_2$ of a transformed vector.) A more general term would be just "regression", or perhaps "M estimation". Dec 17, 2016 at 6:11
• @Repmat this gives a bound perhaps: Top hit is a 1921 astronomy paper using the term consistent with the "vs. weighted" sense. Dec 17, 2016 at 6:27