# What is a scoring rule for binary classification that is not dependent on the "difficulty" of classification?

Consider a model that predicts the probability of some binary event $$Y$$ (potentially given some features $$X$$). Denote the estimated probability of $$Y$$ occurring as $$\hat{p}$$. One possible choice for a (proper) scoring rule to evaluate $$\hat{p}$$ is the logistic scoring rule:

$$-\left(Y\log\left(\hat{p}\right) + (1 - Y)\log\left(1 - \hat{p}\right)\right)$$

The issue I have is that the value of the expected score when making an optimal decision is dependent on the true $$P(Y) = p$$. Let's say $$p = .9$$, then the optimal $$\hat{p} = .9$$, which minimizes the expected score, which in this case is about .325.

Now, let's say $$p = .5$$. In this case, optimal $$\hat{p} = .5$$, leading to an expected score of .693, more than double the optimal score in the previous case.

I have a Bayesian regression model, which I first train on a few hundred data points. I then use the resulting posteriors to generate $$\hat{p}$$s for the next few hundred data points (call this Dataset A). Then, I train the model on both the original training set AND Dataset A, and generate $$\hat{p}$$s for another few hundred data points (call this Dataset B). All 3 datasets follow each other in time.

The average logistic loss is much lower on Dataset A than on Dataset B. The naïve interpretation of this is that the model is somehow "better" at estimating probabilities on Dataset A than on Dataset B, and that somehow the model did not learn anything and/or there was such a distribution shift between A and B, that the model "became worse."

Yet what I believe is happening is that a lot of the points in Dataset A were simply easier to predict than in Dataset B, i.e., the true $$p$$s were more often closer to 0 or 1 in Dataset A, and more often closer to .5 in Dataset B. Thus the average logistic loss is higher in Dataset B, not because the model became a worse model, but because the problem became harder.

How to reconcile this? I would like some metric that tells me how "good" my model is at estimating these probabilities, without being dependent on the underlying distribution of true probabilities. Does one exist?

• Possible duplicate (granted, without an answer)
– Dave
Commented Feb 6 at 22:07
• Your problem seems to be similar to this one: both are about how to find out whether your model has reached the limit of predictability. It also ties into the difficulty of comparing a model's quality between different data sets, this is related. Therefore my recommendation would be to look at the improvement of your model over the performance of a simple benchmark model, see here, which to a degree quantifies your dataset's predictability. Commented Feb 7 at 7:30

Such a scoring rule cannot exist if you want it to be strictly proper. And if it's not strictly proper, it will be useless for your application.

We usually want scoring rules to be (strictly) proper since it implies that predicting the true underlying probability of the event will give the lowest score/loss on average. More precisely, in the binary case the scoring rule $$S : [0,1] \times \lbrace 0,1 \rbrace \to \mathbb{R}$$ is proper if $$\mathbb{E}_{Y\sim p} \, S(p, Y) \le \mathbb{E}_{Y\sim p} \, S(q, Y)$$ for all $$q, p \in [0,1]$$. It is strictly proper if equality implies $$p=q$$. Well-known examples are the Brier score and the log score.

The expected score function is the mapping $$p \mapsto \mathbb{E}_{Y \sim p} S(p, Y)$$ and it gives the minimal achievable expected score for every probability $$p \in [0,1]$$. For example, for the log score you mentioned the expect score function is simply $$- p \log (p) - (1-p) \log (1-p)$$. Under weak regularity conditions it can be shown that a (strictly) proper scoring rule has a (strictly) convex expected score function. Hence, if you want to have a scoring rule where the expected score function is not dependent on the true $$p$$, then it either fails to be proper (so it doesn't reward truthfulness) or it is constant.

As mentioned in Dave's answer you can look at calibration instead. Or as recommended by Stephan Kolassa, you can deal with this issue by calculating the forecast skill, which results from comparing to a reference model, see for instance this question.

### References

The mentioned result and some illustrations can be found in Gneiting and Raftery's Strictly Proper Scoring Rules, Prediction, and Estimation

• Thank you for this answer. I'm realizing now I should have tagged my answer as "calibration," as that's really what I'm after (and what every probabilistic forecast should be after, IMO). And thanks for the reference in your other comment. Commented Mar 9 at 22:36

I think you’re looking for a measure of calibration, which measures how your predicted probability values reflect reality. That is, when $$\hat p=0.7$$ is predicted, does the event happen $$70\%$$ of the time? If that is the case across the predicted probability values $$(P(y=1\vert \hat p = x) \approx x)$$, then you have good calibration. If that fails to be the case, the calibration is poor (even if there is strong discriminative ability, that is, the classes are fairly easy to distinguish).

The Brier score has a popular decomposition into distinct measures of calibration and discrimination, though it has been shown that every strictly proper scoring rule can be decomposed this way.

• Thanks. I found this paper which describes the decomposition for logistic loss: doi.org/10.1175/2010MWR3229.1. However, it seems to be highly dependent on the choice of "probability categories." I'm struggling to see how these decompositions are defined when every forecast is unique Commented Feb 11 at 19:23
• – Dave
Commented Feb 11 at 19:39
• Thanks for sharing that, @picky_porpoise confirmed what I had suspected: "It looks like you hit the $K=N$ case in your example. In this case RES = UNC and REL is just the Brier score, so the decomposition gives no insights." So to get around that, we can bin the different forecasts via their forecasted probabilities, but now the reliability becomes dependent on our choice of bins (and when comparing two models, which forecasts end up in which bin may be different) Commented Feb 11 at 19:52
• @ischmidt20 You can deal with this by using a reliability definition/diagram which does not depend on binning, but is instead based on isotonic regression. I mention the reference in the answer you are citing from. Commented Mar 6 at 20:00