1
$\begingroup$

[I'm not a mathematician, so please forgive any misuse of terminology]

One way of understanding scoring rules is that they measure the 'distance' between the truth value of a statement, and the probability we assign to that statement. So, if 'it's raining' is true, that statement gets assigned 1. We find the difference between 1 and the probability we've assigned it. Let's suppose the difference is 0.3. We then plug that value into the scoring rule. If it's the Brier score, we'd square 0.3 to get 0.09. This is all well and good, but what happens if a) we'd assigned probability 1 to the true statement and b) we were using the logarithmic scoring rule. Since 1-1=0, and 0 is undefined for natural logarithms ... well, what does happen?

Relatedly, are there other scoring rules which have this feature? The only other scoring rules I'm familiar with are Brier and Spherical distance, neither of which do. But presumably there are other scoring rules besides these? (i.e., which aren't just linear transformations of them).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The logarithmic scoring rule is the most sensitive one. To achieve that it has to have the property that a 'wrong' prediction that is certain causes an infinite error. To avoid that, don't allow your predictions to be as extreme as probabilities of 0 or 1. Many people choose the Brier score to avoid this problem.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks Frank. Two questions: - first, if ln(x) is undefined when x = 0, why assume it returns infinity? Is this merely a common convention? And if so, is it dispensable? - second, presumably there are linear transformations of the logarithmic measure that would avoid this problem? $\endgroup$
    – burt_gellorbsen
    Commented Jan 27, 2018 at 13:49
  • 1
    $\begingroup$ The accuracy score would be $-\infty$ if you predicted a probability of 1.0 for an event that did not occur. This is not a convention; it's the math. There is no fix for that when you use "the" logarithmic scoring rule. There are many other rules available. Search for "proper scoring rule". $\endgroup$
    – Frank Harrell
    Commented Jan 27, 2018 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.