# Calibration of a few binary classifiers is not perfect - why?

I am working on a binary classifier using LightGBM. I try to see the results of the classifiers when changing the costs of false positives and false negatives, still working on the same training and validating datasets. The objective function is defined as following:

def my_scorer(y_true, y_pred):
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

model_gain = loss * tp - gain * fp
max_gain = loss * (fn + tp)

return model_gain / max_gain

def lgbm_scorer(labels, preds):
return 'lgbm_scorer', scorer_collection(labels, (preds > 0.5)), True


As I want to have probabilities as a result of my modelling, I use isotonic regression as a final part of the pipeline.

# sklearn version, for the sake of calibration
bst_ = LGBMClassifier(**search_params, **static_params, n_estimators = 1500)

bst_.fit(X = X_train, y = y_train, sample_weight = TRAIN_WEIGHTS,
eval_set = (X_test, y_test), eval_sample_weight = [TEST_WEIGHTS],
eval_metric = lgbm_scorer,
early_stopping_rounds = 150,
callbacks = [lgb.reset_parameter(learning_rate = lambda current_round: learning_rate_decay(current_round,
base_learning_rate = learning_rate,
decay_power = decay_power))],
categorical_feature = cat_vars)

# Calibrate
calibrated_clf = CalibratedClassifierCV(
base_estimator=bst_,
method = 'isotonic',
cv="prefit"
)
calibrated_clf.fit(X_train, y_train)


search_params are hyperparameters defined individually (one set per model) using Optuna so that the ROC-AUC score is approx. the same for all the models, so that they are comparable.

By only changing variables of customized objective function (loss and gain), I can see that most of the classifiers are perfectly calibrated, but just a few are not - all of those few are below the 'perfectly calibrated' line.

Why has that happened? How come the calibration cannot be perfect - overall and in this scenario?

• Why would you assume from the method to always give perfect results..?
– Tim
Jun 14, 2021 at 15:00
• As explained in the post, I got a perfect calibration for almost all of the combinations apart from a few, hence my question.
– Rafa
Jun 15, 2021 at 7:17
• If you tossed coin 10 times and got 8 heads, would you ask why not 10? Algorithms like the one above don’t always give perfect results. No machine learning algorithm does that.
– Tim
Jun 15, 2021 at 18:12
• Providing some of the plots/scores, or ideally some code and sample data, would be very helpful in diagnosis. Jun 15, 2021 at 20:40
• Thanks @BenReiniger thought that keeping it short and simple for discussing only the theory would be a good idea. Added some code to make my problem more clear.
– Rafa
Jun 16, 2021 at 8:22

Probably those models rank-order the samples poorly; since isotonic regression transforms the scores monotonically, it cannot fix the issue of score-bins having actual rates out of order. To check that, compare the area under their ROC curves (since it depends only on the rank-ordering).

• If I understand you correctly, roc curve is probably different for those that are not as well calibrated - hence if optimized the hyperparameters so that the ROC AUC is the same, then the calibration should be fixed as well?
– Rafa
Jun 16, 2021 at 8:18
• Your update (that AUCs are similar) seems to invalidate this answer. I'll leave it as a general answer, but ... Jun 16, 2021 at 13:38

The approach you are using is inconsistent with optimum decision making. Don't use a "classifier" but instead estimate the probability of the outcome as a function of the predictors. Then when making the ultimate decision you apply utilities/costs/losses to the consequences of all possible decisions to make the decision that maximizes expected utility. Note that in the optimum Bayes decision the utilities are applied at the end, and are not part of the estimation process. See Chapter 19 of BBR for more information.

• If I understand correctly, what you're suggesting is: 1. define a classifier which maximizes e.g. ROC AUC score, 2. define a logistic regression model which optimizes the customized objective function, and which takes the same variables as the one before AND the score from the previous classifier. Is that correct?
– Rafa
Jun 17, 2021 at 13:46
• No, no, no. First of all, don't use language 'develop a classifier' when you are not developing a classifier but are fitting a probability model (the likes of which is necessry for ROC calculation). Second, AUROC is not a gold standard objective criterion. Use the likelihood, penalized likelihood, or a Bayesian approach. Then either avoid making a decision (i.e., make only a prediction) or ascertain the utilities needed to translate the predictions into optimum decision. Jun 17, 2021 at 20:13