What is a scoring rule for binary classification that is not dependent on the "difficulty" of classification?

Question

Consider a model that predicts the probability of some binary event $Y$ (potentially given some features $X$). Denote the estimated probability of $Y$ occurring as $\hat{p}$. One possible choice for a (proper) scoring rule to evaluate $\hat{p}$ is the logistic scoring rule:

$$-\left(Y\log\left(\hat{p}\right) + (1 - Y)\log\left(1 - \hat{p}\right)\right)$$

The issue I have is that the value of the expected score when making an optimal decision is dependent on the true $P(Y) = p$. Let's say $p = .9$, then the optimal $\hat{p} = .9$, which minimizes the expected score, which in this case is about .325.

Now, let's say $p = .5$. In this case, optimal $\hat{p} = .5$, leading to an expected score of .693, more than double the optimal score in the previous case.

I have a Bayesian regression model, which I first train on a few hundred data points. I then use the resulting posteriors to generate $\hat{p}$s for the next few hundred data points (call this Dataset A). Then, I train the model on both the original training set AND Dataset A, and generate $\hat{p}$s for another few hundred data points (call this Dataset B). All 3 datasets follow each other in time.

The average logistic loss is much lower on Dataset A than on Dataset B. The naïve interpretation of this is that the model is somehow "better" at estimating probabilities on Dataset A than on Dataset B, and that somehow the model did not learn anything and/or there was such a distribution shift between A and B, that the model "became worse."

Yet what I believe is happening is that a lot of the points in Dataset A were simply easier to predict than in Dataset B, i.e., the true $p$s were more often closer to 0 or 1 in Dataset A, and more often closer to .5 in Dataset B. Thus the average logistic loss is higher in Dataset B, not because the model became a worse model, but because the problem became harder.

How to reconcile this? I would like some metric that tells me how "good" my model is at estimating these probabilities, without being dependent on the underlying distribution of true probabilities. Does one exist?

Answer 1

Such a scoring rule cannot exist if you want it to be strictly proper. And if it's not strictly proper, it will be useless for your application.

We usually want scoring rules to be (strictly) proper since it implies that predicting the true underlying probability of the event will give the lowest score/loss on average. More precisely, in the binary case the scoring rule $S : [0,1] \times \lbrace 0,1 \rbrace \to \mathbb{R}$ is proper if $$ \mathbb{E}_{Y\sim p} \, S(p, Y) \le \mathbb{E}_{Y\sim p} \, S(q, Y) $$ for all $q, p \in [0,1]$. It is strictly proper if equality implies $p=q$. Well-known examples are the Brier score and the log score.

The expected score function is the mapping $p \mapsto \mathbb{E}_{Y \sim p} S(p, Y)$ and it gives the minimal achievable expected score for every probability $p \in [0,1]$. For example, for the log score you mentioned the expect score function is simply $- p \log (p) - (1-p) \log (1-p)$. Under weak regularity conditions it can be shown that a (strictly) proper scoring rule has a (strictly) convex expected score function. Hence, if you want to have a scoring rule where the expected score function is not dependent on the true $p$, then it either fails to be proper (so it doesn't reward truthfulness) or it is constant.

As mentioned in Dave's answer you can look at calibration instead. Or as recommended by Stephan Kolassa, you can deal with this issue by calculating the forecast skill, which results from comparing to a reference model, see for instance this question.

References

The mentioned result and some illustrations can be found in Gneiting and Raftery's Strictly Proper Scoring Rules, Prediction, and Estimation

Answer 2

I think you’re looking for a measure of calibration, which measures how your predicted probability values reflect reality. That is, when $\hat p=0.7$ is predicted, does the event happen $70\%$ of the time? If that is the case across the predicted probability values $(P(y=1\vert \hat p = x) \approx x)$, then you have good calibration. If that fails to be the case, the calibration is poor (even if there is strong discriminative ability, that is, the classes are fairly easy to distinguish).

The Brier score has a popular decomposition into distinct measures of calibration and discrimination, though it has been shown that every strictly proper scoring rule can be decomposed this way.