# Paradox of Brier skill score of perfectly calibrated output?

Given outcomes, $$y \in \{0,1\}$$ and outputs $$o = f(x) \in \mathbb R, o \in [0,1]$$, I'm interested in the case where the model $$f$$ perfectly models the variable $$Y$$. Since $$Y$$ is Bernoulli, this means

$$P(Y=1\mid X=x)=f(x)\\ P(Y=0\mid X=x)=1-f(x)$$

Thus $$E\left[Y\mid X=x\right]=f(x)$$ And \begin{align} \operatorname{Var}(Y|X=x) &=E\left[Y^2\mid X=x\right]-(E\left[Y\mid X=x\right])^2\\ &=E\left[Y\mid X=x\right]-(E\left[Y\mid X=x\right])^2\\ &=f(x)-f(x)^2=f(x)(1-f(x)) \end{align}

If we simulate data given this scenario, and compute the Brier Skill Score (BSS), we see that:

import numpy as np
import pandas as pd
from plotnine import *

def BSS_simulation(n):
f = np.random.uniform(0, 1, n)
y = np.random.binomial(1, f)
yhat = y.mean()
brier_f = ((y - f)**2).mean()
brier_yhat = ((y - yhat)**2).mean()
bss = 1 - brier_f / brier_yhat
return bss

n = 1000
n_sim = 1000
bss = [BSS_simulation(n) for _ in range(n_sim)]

df_bss = pd.DataFrame({"BSS": bss})
(
ggplot(df_bss, aes(x="BSS"))
+ geom_histogram(bins=30)
+ theme_bw()
+ theme(figure_size=(5, 3))
+ labs(title="Brier skill score simulation", x="BSS", subtitle=f"Mean BSS: {np.mean(bss):.2f}")
)


To me, this result is unexpected. A perfectly calibrated prediction does not, in fact, achieve $$BSS=1$$. However, it is trivial to show as well that the only way to attain $$BSS=1$$ is if $$f(x)=y$$ for every single prediction, i.e., the Brier score is zero for model $$f$$.

I apologize for the very hastily made (and devoid of rigor) derivations below, but I believe we can show that (assuming $$f$$ is uniform and the prevalence of the positive class is $$\bar y = 0.5$$, although I believe it can be more rigorously defined given $$f$$)

$$E[(y-f)^2]=\int (y-f)^2 df = \int f(1-f) df = \left. \frac{3f^2-2f^3}{6}\right|_{f=0}^{f=1}=\frac{3-2}{6}=1/6$$

And

$$E[(y-\bar y)^2]=\bar y (1 - \bar y) = 1/2 \cdot 1/2 = 1/4$$

And then the Skill score becomes

$$\text{BSS} = 1 - \frac{1/6}{1/4} = 1 - \frac23=\frac13$$

Does it imply that when we completely capture the data generating process the Brier skill is still far from one?

Brier skill score is equal to $$1-\dfrac{BS}{BS_{ref}}$$, where the numerator is the model Brier score and the denominator is the Brier score of a reference model (a baseline model or some kind of benchmark).

Therefore, Brier skill score equals $$1$$ if, and only if, the model Brier score is $$0$$. This only happens when the predicted probability values are, correctly, zero and one.

Brier score addresses both calibration and discrimination, so the fact that there is perfect calibration is nice, but the Brier score is nonzero if there is not perfect separation of the two categories.

• I have to digest the implications further, but I do agree with this answer. Apr 19 at 10:44
• @Firebug Maybe think of it as being analogous to $R^2$.
– Dave
Apr 19 at 10:46
• Yeah, I'm aware, but I find the fact that when you get the probabilities exactly right the BSS is still so far from 1 suboptimal, because of the lack of resolution/discrimination. Obviously, you can get the calibration correct and still have a bad BSS, but in my example the relation is perfect. Apr 19 at 10:53

The Brier skill score is designed to assess the forecast skill of all possible forecasts and data generating processes. As Dave writes, it equals 1 if, and only if the prediction is perfect, i.e. correctly zero or one.

You present an example where such a perfect prediction is not achievable, but the Brier skill score compares all predictions, even the impossible ones (and also across different datasets). In reality we will never know how much the uncertainty can be reduced, so we don't know the lower limit of the Brier score for the underlying data-generating process. The simplest way is then to not exclude perfect prediction and accept that the skill could go to 1, even though it probably never will for 'real' models/forecasts.

Using the usual three term decomposition, the BSS can be written as

\begin{align} \text{BSS}&= 1-\frac{\operatorname{BS}_f(f,y)}{\operatorname{BS}_0(f,y)}= 1-\frac{\text{REL}-\text{RES}+\text{UNC}}{\text{UNC}}\\&= \frac{\text{UNC}-\text{REL}+\text{RES}-\text{UNC}}{\text{UNC}}= \frac{\text{RES}-\text{REL}}{\text{UNC}} \end{align}

In the example I presented $$\text{REL}=0$$, denoting perfect calibration, i.e., $$\bar f_k = \bar o_k$$. Assuming a fixed number of $$N_k$$ outputs $$f_k$$, each occurring $$n_k$$ times, with $$\bar o_k$$ prevalence per output, and the overall prevalence of the positive class being $$\bar o$$:

\begin{align} \text{BSS}=\frac{\text{RES}}{\text{UNC}}=\frac{\frac{1}{n}\sum_{k=1}^{N_k}n_k \left(\bar{o}_k-\bar o\right)^2}{\bar o(1-\bar o)} \end{align}

The only way to achieve perfect skill is if resolution cancels uncertainty.