# The intuition behind the different scoring rules

Consider the three scoring rules in the case of a binary prediction:

1. Log: sum(log(ifelse(outcome, probability, 1-probability))) / n
2. Brier: sum((outcome-probability)**2) / n
3. Sphere: sum(ifelse(outcome, probability, 1-probability)/sqrt(probability**2+(1-probability)**2)) / n

What is the intuition behind them? When should I use one and not the other? I am especially interested in the case of low prevalence (e.g., 0.1%).

PS. This is to evaluate the results from my calibration algorithm which I asked about before.

• possible duplicate of Justifying and choosing a proper scoring rule
– sds
Commented Apr 24, 2015 at 20:31
• do you think your own post is a duplicate? As I read the linked thread, it does not (currently) answer all the questions I understand in your Q here. I would not vote to close, as I would be interested in answers to your questions (+1 from before). But you can always delete your own Q, if you want. Commented Apr 24, 2015 at 21:07
• @gung: I would love to see an answer too, but the referenced question and its answer is highly related and I wanted to point that out. I think a "possible dupe" is a good way, especially since you clearly indicated your disagreement (thank you!) and thus made the actual closing unlikely. :-)
– sds
Commented Apr 24, 2015 at 21:27
• You can simply add a comment to your Q w/ a link saying that it is related or may also be of interest to readers. That would accomplish what you set out to do here. I would not flag your Q for closing as a duplicate. Commented Apr 24, 2015 at 23:17
• Regarding only 1. the intuition is that it is the log likelihood function for a binary outcome Y which we know has certain optimality properties when maximized to fit statistical models. Commented Nov 15, 2020 at 11:52

One place where log scoring may be inappropriate: the comparison of human forecasters (who may tend to overstate their confidence).

Log scoring strongly penalizes very overconfident wrong predictions. A wrong prediction that was made with 100% confidence gets an infinite penalty. For example, suppose a commentator says "I am 100% sure that Smith will win the election," and then Smith loses the election. Under log scoring, the average score of all the commentator's predictions is now permanently stuck at $$-\infty$$, the worst possible. It should be possible to distinguish that somebody who has made a single wrong 100% confidence prediction is a better forecaster than somebody who makes them all the time.

• I would say that "over-penalizing over-confidence" is a feature, not a bug.
– sds
Commented Dec 20, 2020 at 18:44

# Log

The expected surprisal of the prediction when we discover the actual value.

# Brier

$L^2$, RMSE, OLS.

However, the fact that $p=2$ is the only value which turns the $L^p$ norm into proper scoring rule detracts from this intuition.

# Sphere

The cosine of the angle between the prediction vector $(p,1-p)$ and the outcome vector (0,1) or (1,0).

Note that the angle itself is not a proper scoring rule, which also detracts from the intuition.

• Although you answered your own question, your answer is incoherent as a response to your interesting question. For example, "Log: the expected surprisal of the prediction when we discover the actual value" is incoherent as an answer to the question, "What is the intuition behind them? When should I use one and not the other?" Commented Nov 11, 2020 at 7:00
• @Tripartio: do you have a better answer?
– sds
Commented Nov 11, 2020 at 11:46
• No I don't. I upvoted the question because I would really like to learn the answer. However, I posted a related question: stats.stackexchange.com/questions/495935/… Commented Nov 11, 2020 at 13:03