7
$\begingroup$

Who first used "early stopping" to refer to a form of regularization by stopping training before convergence? I have attempted to search myself but I am not sure how to find the answer.

Was the term ever used in stats before neural networks for example?

$\endgroup$
6
$\begingroup$

It depends on how deep you want to go. Abraham Wald was studying sequential testing and stopping criteria in 1945. It was further studied in the 1960 and 1970s (cf Ferguson, 1989). In 1989 Pocock and Hughes described using early stopping for clinical trials (I'm not sure if this is the earliest paper like this). As about first citations of using it in machine learning, the paper by Morgan and Bourlard (1990) is sometimes quoted, for example by Prechelt (1997) or Finoff et al (1993).

$\endgroup$
8
  • $\begingroup$ I looked at the 1945, 46, 89 and 90 papers and didn't find the phrase "early stopping". It's possible my pdf viewer can't search for the term but is it in those papers? $\endgroup$ – Anush Jun 16 at 15:11
  • 4
    $\begingroup$ Just because the concept was invented/discovered doesn't mean the name was solidified on day 1. (Signified/signifier...) $\endgroup$ – Arya McCarthy Jun 16 at 15:26
  • $\begingroup$ Pocock has a paper (BMJ 1992;305:235-40) entitled "When to stop a clinical trial" which is entirely about the ethics of early stopping. Without being unkind to the statistical expertise of the BMJ and its readers it is not a specialist statistical outlet so I suspect these ideas were around for several years before that and likely before the 1989 reference you quote. $\endgroup$ – mdewey Jun 17 at 13:43
  • 2
    $\begingroup$ @mdewey (and I suppose partly @Tim) I think Pocock is referring to a different meaning of "early stopping". The OP is asking about stopping during iterative optimization, before the parameters' values have converged. $\endgroup$ – Arya McCarthy Jun 17 at 14:07
  • 1
    $\begingroup$ @AryaMcCarthy yes, OP seems to be referring to early stopping in machine learning. As I said in the first sentence, what you consider as the early references to early stopping would depend on how deep you want to go with tracking it. The ML early stopping is an descendant of the ideas and studies on early stopping in other areas. $\endgroup$ – Tim Jun 17 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.