Calibration of a few binary classifiers is not perfect - why?

Question

I am working on a binary classifier using LightGBM. I try to see the results of the classifiers when changing the costs of false positives and false negatives, still working on the same training and validating datasets. The objective function is defined as following:

def my_scorer(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    
    model_gain = loss * tp - gain * fp
    max_gain = loss * (fn + tp)
    
    return model_gain / max_gain

def lgbm_scorer(labels, preds):
    return 'lgbm_scorer', scorer_collection(labels, (preds > 0.5)), True

As I want to have probabilities as a result of my modelling, I use isotonic regression as a final part of the pipeline.

# sklearn version, for the sake of calibration
bst_ = LGBMClassifier(**search_params, **static_params, n_estimators = 1500)

bst_.fit(X = X_train, y = y_train, sample_weight = TRAIN_WEIGHTS,
         eval_set = (X_test, y_test), eval_sample_weight = [TEST_WEIGHTS],
         eval_metric = lgbm_scorer,
         early_stopping_rounds = 150, 
         callbacks = [lgb.reset_parameter(learning_rate = lambda current_round: learning_rate_decay(current_round, 
                                                                                                    base_learning_rate = learning_rate,
                                                                                                    decay_power = decay_power))],
         categorical_feature = cat_vars)

# Calibrate 
calibrated_clf = CalibratedClassifierCV(
    base_estimator=bst_,
    method = 'isotonic',
    cv="prefit"
)
calibrated_clf.fit(X_train, y_train)

search_params are hyperparameters defined individually (one set per model) using Optuna so that the ROC-AUC score is approx. the same for all the models, so that they are comparable.

By only changing variables of customized objective function (loss and gain), I can see that most of the classifiers are perfectly calibrated, but just a few are not - all of those few are below the 'perfectly calibrated' line.

Why has that happened? How come the calibration cannot be perfect - overall and in this scenario?

Answer 1

Probably those models rank-order the samples poorly; since isotonic regression transforms the scores monotonically, it cannot fix the issue of score-bins having actual rates out of order. To check that, compare the area under their ROC curves (since it depends only on the rank-ordering).

Answer 2

The approach you are using is inconsistent with optimum decision making. Don't use a "classifier" but instead estimate the probability of the outcome as a function of the predictors. Then when making the ultimate decision you apply utilities/costs/losses to the consequences of all possible decisions to make the decision that maximizes expected utility. Note that in the optimum Bayes decision the utilities are applied at the end, and are not part of the estimation process. See Chapter 19 of BBR for more information.