1
$\begingroup$

I have been using sklearn's CalibrationDisplay and think it is pretty cool. One thing I am wondering, though, is how I could potentially take that curve and make it an interpretable score. For example, here is a simple model and calibration curve:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.calibration import CalibrationDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=100000, n_features=10, n_classes=2, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

lr = LogisticRegression()

lr.fit(X_train, y_train)

display = CalibrationDisplay.from_estimator(estimator=lr, X=X_test, y=y_test, n_bins=10)

This curve is great for showing calibration, but I want to take it one step further. Using the prob_pred and prob_true attributes returned by from_estimator(), could I take the absolute value of the mean difference between the bins to get an overall calibration percentage error?

prob_pred
array([0.03257269, 0.1443675 , 0.24642089, 0.34935094, 0.45096792,
       0.55000066, 0.65085682, 0.75263477, 0.8547215 , 0.96749284])
prob_true
array([0.03235031, 0.15408416, 0.24418605, 0.34750911, 0.43198091,
       0.56077016, 0.67623421, 0.75992063, 0.85426829, 0.96590106])
np.absolute((display.prob_pred - display.prob_true).mean())
0.002781836006191922

And could this value be interpreted as something similar to: "on average there is an expected 0.278% difference between a given positive prediction's predicted probability and the expected true probability of a positive outcome for said observation", or something similar?

Again, I am very curious as to how I can generate a numeric calibration score from sklearn's CalibrationDisplay and I think what I came up with is a decent proxy? Would love to know more. Thanks!

Edit: my idea came from this excerpt in the documentation: "Well calibrated classifiers are probabilistic classifiers for which the output of predict_proba can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that for the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class."

$\endgroup$
1
  • $\begingroup$ Brier score can be decomposed to give a calibration component. I wonder if that might be what you seek, though +1 for the interesting question! $\endgroup$
    – Dave
    Dec 15, 2022 at 23:00

1 Answer 1

0
$\begingroup$

In the graph produced by the code in the original question, the hope is for the plotted points to follow the line $y=x$. That is, we want the probabilities predicted by the model to match up with the true probabilities of event occurrence (a reasonable desire, I believe).

A standard way to check the deviation of predictions $\hat y$ and true values $y$ is with the $R^2$. Therefore, such a statistic might prove useful here.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

If the predicted and actual probabilities match perfectly, your score will be a perfect $1$. As predictions deviate more and more from the true probability, the score will get worse and worse.

Another approach, granted, not derived from the graph, is that Brier score can be decomposed into measures of model calibration and discrimination (ability to distinguish between the categories), the latter of which might be of interest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.