Correct way to interpret sklearn's calibration error and generate a numeric calibration loss/score

Question

I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example, here is a simple model and calibration curve:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.calibration import CalibrationDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=100000, n_features=10, n_classes=2, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

lr = LogisticRegression()

lr.fit(X_train, y_train)

display = CalibrationDisplay.from_estimator(estimator=lr, X=X_test, y=y_test, n_bins=10)

This curve is great for showing calibration, but I want to take it one step further. Using the prob_pred and prob_true attributes returned by from_estimator(), could I take the absolute value of the mean difference between the bins to get an overall calibration percentage error?

prob_pred
array([0.03257269, 0.1443675 , 0.24642089, 0.34935094, 0.45096792,
       0.55000066, 0.65085682, 0.75263477, 0.8547215 , 0.96749284])
prob_true
array([0.03235031, 0.15408416, 0.24418605, 0.34750911, 0.43198091,
       0.56077016, 0.67623421, 0.75992063, 0.85426829, 0.96590106])
np.absolute((display.prob_pred - display.prob_true).mean())
0.002781836006191922

And could this value be interpreted as something similar to: "on average there is an expected 0.278% difference between a given positive prediction's predicted probability and the expected true probability of a positive outcome for said observation", or something similar?

Again, I am very curious as to how I can generate a numeric calibration score from sklearn's CalibrationDisplay and I think what I came up with is a decent proxy? Would love to know more. Thanks!

Edit: my idea came from this excerpt in the documentation: "Well calibrated classifiers are probabilistic classifiers for which the output of predict_proba can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that for the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class."

Answer 1

In the graph produced by the code in the original question, the hope is for the plotted points to follow the line $y=x$. That is, we want the probabilities predicted by the model to match up with the true probabilities of event occurrence (a reasonable desire, I believe).

A standard way to check the deviation of predictions $\hat y$ and true values $y$ is with the $R^2$. Therefore, such a statistic might prove useful here.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

If the predicted and actual probabilities match perfectly, your score will be a perfect $1$. As predictions deviate more and more from the true probability, the score will get worse and worse.

Another approach, granted, not derived from the graph, is that Brier score can be decomposed into measures of model calibration and discrimination (ability to distinguish between the categories), the latter of which might be of interest.