A friend of mine recently asked what is so ordinary, about ordinary least squares. We did not seem to get anywhere in the discussion. We both agreed that OLS is special case of the linear model, it has many uses, is well know, and is a special case of many other models. But is this really all?
Therefore I would like to know:
Least squares in $y$ is often called ordinary least squares (OLS) because it was the first ever statistical procedure to be developed circa 1800, see history. It is equivalent to minimizing the $L_2$ norm, $||Y-f(X)||_2$. Subsequently, weighted least squares, minimization of other norms (e.g., $L_1$), generalized least squares, M Estimation, bivariate minimization (e.g., Deming regression), non-parametric regression, maximum likelihood regression, regularization (e.g., Tikhonov, ridge) and other inverse problem techniques and multiple other tools were developed. There is still controversy over who first applied it, Gauss or Legendre (see link). The term "ordinary" (implying in $y$) was obviously added to "least squares" only after so many alternative methods arose that the (still most) popular OLS needed to be differentiated from the plethora of other minimizations that became available. The word ordinary is often used in mathematical jargon as a synonym of simple. For example, consider the phrase ordinary differential equations. When exactly adding ordinary$+$least squares occurred would be hard to track down since that occurred when it became natural or obvious to do so. EDIT (for @alexis): Early 20th century I would presume, @GeoMatt22 says 1921 in comments below, but the mathematical meaning of ordinary (sive, vanilla) predates that and certain differential equations were called ordinary in the late 18th century as per The History of Differential Equations, 1670--1950.
