I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example, here is a simple model and calibration curve:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.calibration import CalibrationDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=100000, n_features=10, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
lr = LogisticRegression()
lr.fit(X_train, y_train)
display = CalibrationDisplay.from_estimator(estimator=lr, X=X_test, y=y_test, n_bins=10)
This curve is great for showing calibration, but I want to take it one step further. Using the prob_pred
and prob_true
attributes returned by from_estimator()
, could I take the absolute value of the mean difference between the bins to get an overall calibration percentage error?
prob_pred
array([0.03257269, 0.1443675 , 0.24642089, 0.34935094, 0.45096792,
0.55000066, 0.65085682, 0.75263477, 0.8547215 , 0.96749284])
prob_true
array([0.03235031, 0.15408416, 0.24418605, 0.34750911, 0.43198091,
0.56077016, 0.67623421, 0.75992063, 0.85426829, 0.96590106])
np.absolute((display.prob_pred - display.prob_true).mean())
0.002781836006191922
And could this value be interpreted as something similar to: "on average there is an expected 0.278% difference between a given positive prediction's predicted probability and the expected true probability of a positive outcome for said observation", or something similar?
Again, I am very curious as to how I can generate a numeric calibration score from sklearn's CalibrationDisplay and I think what I came up with is a decent proxy? Would love to know more. Thanks!
Edit: my idea came from this excerpt in the documentation: "Well calibrated classifiers are probabilistic classifiers for which the output of predict_proba can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that for the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class."