Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When I introduce concepts to my students, I often find it fun to tell them where the terminology originates ("regression", for example, is a term with an interesting origin). I haven't been able to turn up the history/background of the term "regularization" in statistical/machine learning.
So, what is the origin of the term regularization?
Similar to Matthew Gunn's contribution, this is also not really an answer, but more of a plausible candidate.
I also first heard of the term "regularization" in the context of Tikhonov Regularization, and in particular in the context of (linear) inverse problems in geophysics. Interestingly, while I had thought that was likely due to me area of study (i.e. see my username), apparently Tikhonov actually did much of his work in that area!
My hunch is that the modern "regularization" approach likely did originate with Tikhonov's work. Building on this speculation, my contribution here has two parts.
The first part is (armchair-)historical in nature (based on perusing paper titles and my own prior biases!). While the 1963 paper Solution of incorrectly formulated problems and the regularization method appears to be the first use of the term "regularization", I would not be too certain that this is true. This reference is cited in Wikipedia as
Tikhonov, A. N. (1963). "О решении некорректно поставленных задач и методе регуляризации". Doklady Akademii Nauk SSSR. 151: 501–504. Translated in "Solution of incorrectly formulated problems and the regularization method". Soviet Mathematics. 4: 1035–1038.
giving an impression that Tikhonov himself wrote at least some of this work in Russian originally, so the phrase "regularization" could have been coined by a later translator. [UPDATE: No, "регуляризации" = regularization, see comment by Cagdas Ozgenc.] Moreover, this work appears to be part of a continuous line of research conducted by Tikhonov over a much longer time. For example the paper
Tikhonov, Andrey Nikolayevich (1943). "Об устойчивости обратных задач" [On the stability of inverse problems]. Doklady Akademii Nauk SSSR. 39 (5): 195–198.
shows that he was engaged in the same general topic at least 20 years prior. However this timeline suggests that probably the inverse-problem work started much closer to 1963 than to 1943.
[UPDATE: This translation of the 1943 paper shows that the terminology for "regularity" of was here used to refer to the "stability of the inverse problem (or the continuity of the inverse mapping)".]
The second part of my contribution is a hypothesis on how "regularization" may have been originally intended in this context. Quite commonly "regular" is used as a synonym for "smooth", particular in describing curve and/or surface geometry. In most geophysics applications, the desired solution is some gridded estimate of a spatially distributed field, and Tikhonov regularization is used to impose a smoothness prior.
(The Tikhonov matrix will typically be a discrete spatial derivative operator, akin to PDE matrices, vs. the identity matrix of ridge regression. This is because for these grids/forward models, the null-space of the forward-model matrix tends to include things like "checkerboard modes" that will pollute the results unless penalized; similar to this).
Update: These issues are illustrated in my answer here.
Summary
(*Based on the updated quote from the 1943 paper, this phrasing appears to be true ... but for the wrong reason! The relevant "map" was not between grid and field, $u[x]=F[\theta]$, but the inverse mapping from a forward model $\theta=F^{-1}[u]$.)
This is part answer, part long comment. An incomplete list of candidates:
Tikhonov, Andrey. "Solution of incorrectly formulated problems and the regularization method." Soviet Math. Dokl.. Vol. 5. 1963. Tikhonov is known for Tikhonov regularization (also known as ridge regression).
There's a concept of regularization in physics that goes back at least to the 1940s, but I don't see any connection with Tikhonov regularization? (I'm not a physicist though.)
Engineering texts speak of regularization of a river (to improve navigation) going back at least to the 1880s.
Searching through http://books.google.com, I don't see widespread use of the term "regularization" until the 1970s, when it starts showing up again and again and again in the context of mathematics and physics books.
Most simply, the term survived the natural evolution of scientific terms because it captures the core goal of the technique: from a bunch of solutions to an ill-posed problem, it chooses the solutions which are regular, that is,
according to rule
(free dictionary's definition)
This is also used in common language for designing a smooth surface in carpentry for instance. Similarly, the solutions of a regression problem will look more regular if the rule is to minimize total variation (TV) of unsmooth bits of the reconstructed signal (as measured by the total energy of the gradient for instance).
The term became wide-spread because it is very generic: anyone can define its one rule, from TV to L1-norm measures or by using the $\ell_0$ pseudo-norm! As such, the rule may play a similar role as the prior in Bayesian statistics.
