11
$\begingroup$

I'm trying to track down the original reference for the logarithmic loss (logarithmic scoring rule, cross-entropy), usually defined as:

$$L_{log}=y_{true} \log(p) + (1-y_{true}) \log(1-p)$$

For the Brier score for example there is the Brier (1950) article. Is there such a reference for the log-loss?

$\endgroup$
7
  • $\begingroup$ shouldn't cross-entropy have two different distributions, not the same ($p$)? $\endgroup$
    – develarist
    Oct 27 '20 at 19:36
  • 1
    $\begingroup$ The metric is defined in full in the link: scikit-learn.org/stable/modules/model_evaluation.html#log-loss $\endgroup$
    – Gabriel
    Oct 27 '20 at 19:38
  • $\begingroup$ Closely related: stats.stackexchange.com/questions/31985/… $\endgroup$
    – Sycorax
    Oct 27 '20 at 21:18
  • $\begingroup$ @Sycorax: Does it not qualify as a plain duplicate? I feel like I might be missing the difference if there is any. $\endgroup$
    – user541686
    Oct 28 '20 at 7:27
  • $\begingroup$ @user541686 My answer writes about a slim distinction that I perceive between the two. I can see a case made for either closing as a duplicate or letting this more specific variation on the theme stand on its own. $\endgroup$
    – Sycorax
    Oct 28 '20 at 14:14
11
$\begingroup$

The earliest I have been able to find is

Good, I. J. “Rational Decisions.” Journal of the Royal Statistical Society. Series B (Methodological), vol. 14, no. 1, 1952, pp. 107–114. JSTOR, www.jstor.org/stable/2984087

Look at section 8, "Fair Fees":

By itself $\log p_1$ (or $\log(1 - p_1)$) is a measure of the merit of a probability estimate

I found this reference in Gneiting & Raftery (2007, JASA), who write that "This scoring rule dates back at least to Good (1952)", suggesting that they already did a similar search for original sources.

$\endgroup$
1
  • $\begingroup$ Selecting this answer as it adheres more closely to the modern definition of log-loss. Thank you! $\endgroup$
    – Gabriel
    Oct 27 '20 at 20:41
9
$\begingroup$

If we view minimizing cross entropy as equivalent to maximizing the log-likelihood of the same model, then I believe we can go as far back as RA Fisher. This places the date between 1912 and 1922, depending on how well-developed you wish the theory to be; see discussion in John Aldrich "R. A. Fisher and the Making of Maximum Likelihood 1912 – 1922" Statistical Science. 1997, Vol. 12, No. 3, 162-176

We also have this closely related thread: Definition and origin of “cross entropy” which appears to use the term "cross entropy" in the broad sense of a family of probabilistic losses, instead of the sense used in this post, as jargon for a specific loss for a model of binary data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.