I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example, here is a simple model and calibration curve:
import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import make_classification from sklearn.calibration import CalibrationDisplay from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split X, y = make_classification(n_samples=100000, n_features=10, n_classes=2, random_state=42) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) lr = LogisticRegression() lr.fit(X_train, y_train) display = CalibrationDisplay.from_estimator(estimator=lr, X=X_test, y=y_test, n_bins=10)
This curve is great for showing calibration, but I want to take it one step further. Using the
prob_true attributes returned by
from_estimator(), could I take the absolute value of the mean difference between the bins to get an overall calibration percentage error?
prob_pred array([0.03257269, 0.1443675 , 0.24642089, 0.34935094, 0.45096792, 0.55000066, 0.65085682, 0.75263477, 0.8547215 , 0.96749284]) prob_true array([0.03235031, 0.15408416, 0.24418605, 0.34750911, 0.43198091, 0.56077016, 0.67623421, 0.75992063, 0.85426829, 0.96590106]) np.absolute((display.prob_pred - display.prob_true).mean()) 0.002781836006191922
And could this value be interpreted as something similar to: "on average there is an expected 0.278% difference between a given positive prediction's predicted probability and the expected true probability of a positive outcome for said observation", or something similar?
Again, I am very curious as to how I can generate a numeric calibration score from sklearn's CalibrationDisplay and I think what I came up with is a decent proxy? Would love to know more. Thanks!
Edit: my idea came from this excerpt in the documentation: "Well calibrated classifiers are probabilistic classifiers for which the output of predict_proba can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that for the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class."