Why is LogLoss preferred over other proper scoring rules?

Question

It seems anytime people care about estimating probabilities accurately they choose LogLoss as the evaluation metric. But there are many other evaluation metrics which will prefer accurate estimation of probabilities (and not only ranking or performance at some cut-off). Including RMSE (on the continuous model score vs 0/1 actual). Obviously LogLoss penalizes over confidence very heavily, Why is this desirable? When is it not?

Answer 1

Arguments for the log score

On the one hand, as kjetil b halvorsen writes, the log loss is just a reformulation of the log likelihood, which statisticians are very used to maximizing, so it is simply very natural as a KPIs. (A somewhat more common convention is to minimize the score, in which case, one takes the negative log of predicted probabilities, but the same point still applies.)

On the other hand, in the single-class classification case, Merkle & Steyvers (2013, Decision Analysis) point out that the log score is just one member of an entire family of strictly proper scoring rules, which are indexed by two parameters $\alpha\geq 0$ and $\beta\geq 0$. Particular values of $\alpha$ and $\beta$ can be set based on the cost $c$ of misclassifications (based, in turn, on comparing probabilistic predictions to a threshold). Smaller values of $\alpha+\beta$ correspond to higher uncertainty in $c$... and the log score just happens to be the member of the family with $\alpha=\beta=0$. So at least in this classification case, you could say that the log score is a reasonable choice (within this family of scoring rules) that corresponds to the highest uncertainy or agnosticity in the misclassification cost.

On the third hand, Benedetti (2010, Monthly Weather Review) considers three properties a scoring rule should have:

• it should be additive when adding a new event
• it should only depend on the probabilities assigned to events that actually occur and can be observed ("locality")
• and it should be proper (more strongly, Benedetti requires differentiability in predictions and a derivative of zero at the true probabilities)

Benedetti (2010) then proceeds to show that the log loss is the only scoring rule that satisfies these conditions in the case of finitely many possible events.

(To be honest, I don't quite follow Benedetti's derivation; specifically, I don't see how he arrives at equation (7). But I'll put this edit in here as a pointer so smarter people than me can look at the paper.)

Benedetti (2010) then explores connections to information theory and Kullback-Leibler divergence between the probabilistic prediction and the actual outcome distribution. He draws attention to one disadvantage of the Brier score: it depends on probabilities predicted for unobserved events and thus violates the locality requirement. Specifically, assume we have $R=3$ possible events and two different probabilistic predictions, $(0.2,0.4,0.4)$ and $(0.2,0.3,0.5)$. Suppose further that the first event actually occurs. Note that both predictions assign the same probability of $0.2$ to this event. Locality would require both predictions' scores to be identical, since they only differ on predicted probabilities for unobserved events. However, the multi-category Brier score for the first prediction is

$$ (1-0.2)^2+0.4^2+0.4^2 = 0.96 $$

whereas the score for the second prediction is

$$ (1-0.2)^2+0.3^2+0.5^2 = 0.98. $$

However, as Benedetti (2010) points out, the Brier score is a second-order approximation to the logarithmic skill score, which explains some of its appeal.

Finally, one more argument for the log loss that I'm taking from Benedetti (2010, p. 208): if an event occurs which we had predicted to be completely impossible, $\hat{p}=0$, then the log loss is infinite, with no chance of being "saved" by other better predictions. Thus, using the log loss truly forces us to consider the possibility of extremely rare events and not just sweep them under the table. The Brier score, in contrast, is much more relaxed about observing events predicted to be impossible.

For instance, Jewson (2004, arXiv:physics/0401046v1) gives the following example: assume a simple two-class prediction situation. The event occurs with a true probability of $p=0.1$. We have two competing predictions: the first is that the event is impossible, $\hat{p}_1=0$, the second overestimates the true probability, $\hat{p}_2=0.25$. Then the expected Brier score for the first prediction is

$$ 0.1\times 1^2+0.9\times 0^2 = 0.1 $$

whereas the expected Brier score for the second prediction is

$$ 0.1\times (1-0.25)^2+0.9\times 0.25^2 = 0.1125. $$

So the Brier score would actually prefer the first prediction, which is completely off base in that it considers an event with a $0.1$ probability of occurring as completely impossible. This does not make intuitive sense.

Arguments for the Brier score

Of course, the Brier score also has advantages. For instance, the log score explodes if we observe an event that we thought would be impossible, because we then take the log of zero. To some, that's a feature (see above), to others, it's a bug. The Brier score will still be defined if an "impossible" event occurs.

The Brier score is conceptually very close to the Mean Squared Error, and can in fact be expressed as such (between a vector of probabilistic predictions and a 0-1 vector of which class actually occurred). This is easy to understand.

Selten (1998, Experimental Economics) offers four axioms we could require a scoring rule to fulfill:

• it should be symmetric if classes are reordered
• adding a class with zero predicted and true probability should not change the score
• if the true class probabilities are $p=(p_1, \dots, p_k)$ and we predict $\hat{p}=(\hat{p}_1, \dots, \hat{p}_k)$, then the score should be positive (i.e., "bad", see above on conventions about positive and negative orientation) - this is strict properness, which Selten (1998) calls "incentive compatibility"
• if the true class probabilities are $p$ and we predict $\hat{p}$, then the score should be equal to the case where the true probabilities are $\hat{p}$ and we predict $p$ (symmetry; Selten calls this "neutrality")

Selten (1998) then shows that the Brier score is the only one that satisfies these axioms, up to scaling. The log score, of course, violates the fourth requirement, because in general

$$ p\log \hat p \neq \hat p\log p. $$

---

So one way of looking at it is whether we prefer Benedetti's argument that a scoring rule should be "local" (i.e., not be influenced by predicted probabilities for unobserved events), or Selten's argument that it should be symmetric (i.e., give the same result if we exchange the predicted and the true probability vector). In the first case, we should use the log score, in the second case the Brier score.

I personally find Selten's requirement of symmetry (the fourth bullet point in the Brier section above) unnecessary, and I consider the log score explosion a feature and not a bug (see above). Thus, I prefer the log score.