28
$\begingroup$

When I introduce concepts to my students, I often find it fun to tell them where the terminology originates ("regression", for example, is a term with an interesting origin). I haven't been able to turn up the history/background of the term "regularization" in statistical/machine learning.

So, what is the origin of the term regularization?

$\endgroup$
  • 1
    $\begingroup$ My best guess is that regression goes back to Galton. Steve Stigler said that whenever someone claims to be the first they will be found to be wrong. Many people think Efron was the first to coin the term bootstrap in the context of statistics. However Simon claimed to have used it in the early 1960s. There are many different uses of regularization in mathematics and statistics. In Wikipedia: en.wikipedia.org/wk/Regularization_(mathematics) . $\endgroup$ – Michael Chernick Dec 10 '16 at 0:11
  • $\begingroup$ So many methods, so many possible firsts. $\endgroup$ – Michael Chernick Dec 10 '16 at 0:12
  • 3
    $\begingroup$ @MichaelChernick I prefer Tukey's term for bootstrap, "The Shotgun" as it will blow the head off of any problem you have, except that you must put the pieces back together :) $\endgroup$ – AdamO Dec 10 '16 at 0:13
  • 6
    $\begingroup$ I know the term has been in common use since before 1980s. Tikhonov, Arsenin 1977 wrote a text "Solution of Ill-posed Problems" which describes a method for obtaining smoothed solutions to overspecified linear equations, a result which was later found to be equivalent to Ridge Regression (see Girard 1991). I think Tikhonov used the name "Method of Regularization". I further think Hastie abstracted "regularization" in a manner that reflects its modern usage. $\endgroup$ – AdamO Dec 10 '16 at 0:24
  • 5
    $\begingroup$ Terms like "regularization of sequences" have been around in mathematics for a long time (certainly since the 1920s), which has a meaning fairly closely related to the regularization of ill-posed problems. I suspect the use of the word in mathematics would derive from its use in engineering ("regularization of flow" for example). $\endgroup$ – Glen_b Dec 10 '16 at 1:53
18
$\begingroup$

Similar to Matthew Gunn's contribution, this is also not really an answer, but more of a plausible candidate.

I also first heard of the term "regularization" in the context of Tikhonov Regularization, and in particular in the context of (linear) inverse problems in geophysics. Interestingly, while I had thought that was likely due to me area of study (i.e. see my username), apparently Tikhonov actually did much of his work in that area!

My hunch is that the modern "regularization" approach likely did originate with Tikhonov's work. Building on this speculation, my contribution here has two parts.

The first part is (armchair-)historical in nature (based on perusing paper titles and my own prior biases!). While the 1963 paper Solution of incorrectly formulated problems and the regularization method appears to be the first use of the term "regularization", I would not be too certain that this is true. This reference is cited in Wikipedia as

Tikhonov, A. N. (1963). "О решении некорректно поставленных задач и методе регуляризации". Doklady Akademii Nauk SSSR. 151: 501–504. Translated in "Solution of incorrectly formulated problems and the regularization method". Soviet Mathematics. 4: 1035–1038.

giving an impression that Tikhonov himself wrote at least some of this work in Russian originally, so the phrase "regularization" could have been coined by a later translator. [UPDATE: No, "регуляризации" = regularization, see comment by Cagdas Ozgenc.] Moreover, this work appears to be part of a continuous line of research conducted by Tikhonov over a much longer time. For example the paper

Tikhonov, Andrey Nikolayevich (1943). "Об устойчивости обратных задач" [On the stability of inverse problems]. Doklady Akademii Nauk SSSR. 39 (5): 195–198.

shows that he was engaged in the same general topic at least 20 years prior. However this timeline suggests that probably the inverse-problem work started much closer to 1963 than to 1943.

[UPDATE: This translation of the 1943 paper shows that the terminology for "regularity" of was here used to refer to the "stability of the inverse problem (or the continuity of the inverse mapping)".]

The second part of my contribution is a hypothesis on how "regularization" may have been originally intended in this context. Quite commonly "regular" is used as a synonym for "smooth", particular in describing curve and/or surface geometry. In most geophysics applications, the desired solution is some gridded estimate of a spatially distributed field, and Tikhonov regularization is used to impose a smoothness prior.

(The Tikhonov matrix will typically be a discrete spatial derivative operator, akin to PDE matrices, vs. the identity matrix of ridge regression. This is because for these grids/forward models, the null-space of the forward-model matrix tends to include things like "checkerboard modes" that will pollute the results unless penalized; similar to this).

Update: These issues are illustrated in my answer here.


Summary

  1. I also cast my vote for Tikhonov as the originator (likely circa 1963)
  2. The original applications may have been geophysical inverse modeling, so the term "regularization" may refer to making the resulting maps* more smooth, i.e. "regular".

(*Based on the updated quote from the 1943 paper, this phrasing appears to be true ... but for the wrong reason! The relevant "map" was not between grid and field, $u[x]=F[\theta]$, but the inverse mapping from a forward model $\theta=F^{-1}[u]$.)

$\endgroup$
  • 10
    $\begingroup$ 1963 paper contains "regularization" in the Russian version of the title. Unless somebody translated to English and Tikhonov translated the title back to its phonetic reading, it is there on the original. "O rehsnenii nikorrektna postavlennih zadach i metodi regularizatsii" $\endgroup$ – Cowboy Trader Dec 10 '16 at 14:33
14
$\begingroup$

This is part answer, part long comment. An incomplete list of candidates:

  1. Tikhonov, Andrey. "Solution of incorrectly formulated problems and the regularization method." Soviet Math. Dokl.. Vol. 5. 1963. Tikhonov is known for Tikhonov regularization (also known as ridge regression).

  2. There's a concept of regularization in physics that goes back at least to the 1940s, but I don't see any connection with Tikhonov regularization? (I'm not a physicist though.)

  3. Engineering texts speak of regularization of a river (to improve navigation) going back at least to the 1880s.

Searching through http://books.google.com, I don't see widespread use of the term "regularization" until the 1970s, when it starts showing up again and again and again in the context of mathematics and physics books.

$\endgroup$
  • 5
    $\begingroup$ 1. is definitely a candidate. However I don't think that 2. or 3. correspond to the mathematical concept of regularization that the OP is after. Wikipedia agrees in that they have made "regularization (mathematics)" and "regularization (physics)" separate articles. $\endgroup$ – Gordon Smyth Dec 10 '16 at 6:36
3
$\begingroup$

Most simply, the term survived the natural evolution of scientific terms because it captures the core goal of the technique: from a bunch of solutions to an ill-posed problem, it chooses the solutions which are regular, that is,

according to rule

(free dictionary's definition)

This is also used in common language for designing a smooth surface in carpentry for instance. Similarly, the solutions of a regression problem will look more regular if the rule is to minimize total variation (TV) of unsmooth bits of the reconstructed signal (as measured by the total energy of the gradient for instance).

The term became wide-spread because it is very generic: anyone can define its one rule, from TV to L1-norm measures or by using the $\ell_0$ pseudo-norm! As such, the rule may play a similar role as the prior in Bayesian statistics.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.