11
$\begingroup$

I'm trying to track down the original reference for the logarithmic loss (logarithmic scoring rule, cross-entropy), usually defined as:

$$L_{log}=y_{true} \log(p) + (1-y_{true}) \log(1-p)$$

For the Brier score for example there is the Brier (1950) article. Is there such a reference for the log-loss?

$\endgroup$
  • $\begingroup$ shouldn't cross-entropy have two different distributions, not the same ($p$)? $\endgroup$ – develarist Oct 27 at 19:36
  • 1
    $\begingroup$ The metric is defined in full in the link: scikit-learn.org/stable/modules/model_evaluation.html#log-loss $\endgroup$ – Gabriel Oct 27 at 19:38
  • $\begingroup$ Closely related: stats.stackexchange.com/questions/31985/… $\endgroup$ – Sycorax Oct 27 at 21:18
  • $\begingroup$ @Sycorax: Does it not qualify as a plain duplicate? I feel like I might be missing the difference if there is any. $\endgroup$ – user541686 Oct 28 at 7:27
  • $\begingroup$ @user541686 My answer writes about a slim distinction that I perceive between the two. I can see a case made for either closing as a duplicate or letting this more specific variation on the theme stand on its own. $\endgroup$ – Sycorax Oct 28 at 14:14
11
$\begingroup$

The earliest I have been able to find is

Good, I. J. “Rational Decisions.” Journal of the Royal Statistical Society. Series B (Methodological), vol. 14, no. 1, 1952, pp. 107–114. JSTOR, www.jstor.org/stable/2984087

Look at section 8, "Fair Fees":

By itself $\log p_1$ (or $\log(1 - p_1)$) is a measure of the merit of a probability estimate

I found this reference in Gneiting & Raftery (2007, JASA), who write that "This scoring rule dates back at least to Good (1952)", suggesting that they already did a similar search for original sources.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Selecting this answer as it adheres more closely to the modern definition of log-loss. Thank you! $\endgroup$ – Gabriel Oct 27 at 20:41
9
$\begingroup$

If we view minimizing cross entropy as equivalent to maximizing the log-likelihood of the same model, then I believe we can go as far back as RA Fisher. This places the date between 1912 and 1922, depending on how well-developed you wish the theory to be; see discussion in John Aldrich "R. A. Fisher and the Making of Maximum Likelihood 1912 – 1922" Statistical Science. 1997, Vol. 12, No. 3, 162-176

We also have this closely related thread: Definition and origin of “cross entropy” which appears to use the term "cross entropy" in the broad sense of a family of probabilistic losses, instead of the sense used in this post, as jargon for a specific loss for a model of binary data.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.