13
$\begingroup$

I'm trying to track down the original reference for the logarithmic loss (logarithmic scoring rule, cross-entropy), usually defined as:

$$L_{log}=y_{true} \log(p) + (1-y_{true}) \log(1-p)$$

For the Brier score for example there is the Brier (1950) article. Is there such a reference for the log-loss?

Improve this question
$\endgroup$
7
  • $\begingroup$ shouldn't cross-entropy have two different distributions, not the same ($p$)? $\endgroup$
    – develarist
    Commented Oct 27, 2020 at 19:36
  • 1
    $\begingroup$ The metric is defined in full in the link: scikit-learn.org/stable/modules/model_evaluation.html#log-loss $\endgroup$
    – Gabriel
    Commented Oct 27, 2020 at 19:38
  • $\begingroup$ Closely related: stats.stackexchange.com/questions/31985/… $\endgroup$
    – Sycorax ♦
    Commented Oct 27, 2020 at 21:18
  • $\begingroup$ @Sycorax: Does it not qualify as a plain duplicate? I feel like I might be missing the difference if there is any. $\endgroup$
    – user541686
    Commented Oct 28, 2020 at 7:27
  • $\begingroup$ @user541686 My answer writes about a slim distinction that I perceive between the two. I can see a case made for either closing as a duplicate or letting this more specific variation on the theme stand on its own. $\endgroup$
    – Sycorax ♦
    Commented Oct 28, 2020 at 14:14

2 Answers 2

Reset to default
13
$\begingroup$

The earliest I have been able to find is

Good, I. J. “Rational Decisions.” Journal of the Royal Statistical Society. Series B (Methodological), vol. 14, no. 1, 1952, pp. 107–114. JSTOR, www.jstor.org/stable/2984087

Look at section 8, "Fair Fees":

By itself $\log p_1$ (or $\log(1 - p_1)$) is a measure of the merit of a probability estimate

I found this reference in Gneiting & Raftery (2007, JASA), who write that "This scoring rule dates back at least to Good (1952)", suggesting that they already did a similar search for original sources.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Selecting this answer as it adheres more closely to the modern definition of log-loss. Thank you! $\endgroup$
    – Gabriel
    Commented Oct 27, 2020 at 20:41
10
$\begingroup$

If we view minimizing cross entropy as equivalent to maximizing the log-likelihood of the same model, then I believe we can go as far back as RA Fisher. This places the date between 1912 and 1922, depending on how well-developed you wish the theory to be; see discussion in John Aldrich "R. A. Fisher and the Making of Maximum Likelihood 1912 – 1922" Statistical Science. 1997, Vol. 12, No. 3, 162-176

We also have some related threads:

which use the term "cross entropy" in the broad sense of a family of probabilistic losses, instead of the sense used in this post, as jargon for a specific loss for a model of binary data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.