Calibration curve of XGBoost for binary classification

Question

I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated. However, I'm getting a very puzzling result:

xgb_clf = xgb.XGBClassifier(n_estimators=1000, 
                            learning_rate=0.01, 
                            max_depth=3, 
                            subsample=0.8, 
                            colsample_bytree=1, 
                            gamma=1, 
                            objective='binary:logistic', 
                            scale_pos_weight = 10)

y_score_xgb = cross_val_predict(estimator=xgb_clf, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb[:,1], n_bins=10)

It seems like a "nice" (linear) reliability curve, however, the slope is less than 45 degrees.

and here is the classification report:

However, if I do calibration, the resulting curve looks even worse:

calibrated = CalibratedClassifierCV(xgb_clf, method='sigmoid', cv=5)

y_score_xgb_clb = cross_val_predict(estimator=calibrated, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb_clb[:,1], n_bins=10)

What is more strange is that the outputted probabilities now clipped at ~0.75 (I don't get scores higher than 0.75).

Any suggestions / flaws in my approach?

Answer 1

I'm not sure "the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated" is correct: gradient boosting tends to push probability toward 0 and 1. More significantly, you're applying weights (scale_pos_weight=10), which will skew your probabilities higher than the data would suggest.

Because gradient boosting pushes probabilities outward rather than inward, using Platt scaling (method='sigmoid') is generally not the best bet. On the other hand, your original calibration plot does look vaguely like the leftmost part of a sigmoid function. But that explains why your recalibrated scores get cut off at 0.75: fitting a sigmoid onto your calibration plot (which isn't actually what happens, but close enough) will have the right half of the sigmoid cut off.

For expediency, I would first try method='isotonic'. For better understanding, I would suggest shifting scores to account for the weighting you gave, and see where the calibration plot sits then. (The shifting correction is better documented for logistic regression, but see Does down-sampling change logistic regression coefficients? and Convert predicted probabilities after downsampling to actual probabilities in classification .

Finally, sklearn's calibration_curve uses equal-width bins by default, which in an inbalanced dataset is probably not best. You might want to modify it to use equal-size (as in, number of datapoints) bins instead to get a better picture. In particular, I suspect the last two points on your second calibration curve represent very few datapoints, and should be taken with a grain of salt. (In sklearn v0.21, this became easier with the new parameter strategy='quantile'.)

Answer 2

I'm not that familiar with gradient boosting, but I would assume that if you scale your minority class then your model will not be well calibrated. At the end of the day, it has learnt the distribution of the training data which does not reflect reality.

As for CalibratedClassifierCV, from reading the docs it seems that the sigmoid method is not applicable here given your distortion is not sigmoid shaped. Hence, if you have enough data that overfitting is not an issue, then why not try method='isotonic'?

Answer 3

Answer:

No wonder your post-calibration plot looks like this. Look at your reliability curve pre-calibration. The more predicted score grows, the more actual positives it picks up. However, the best your model can do is to extract around 20% of actual positives (when the predicted score is over 0.9). The ideal calibrator would squeeze your probability predictions into [0, 0.2] interval because your model can't do any better. In other words, a great calibrator would map your orange points onto the diagonal line by moving them approximately sideways to the left. You can't expect to get a well-calibrated score of 0.7 ever with this model because it's just not as good.

Additional remarks:

You're using Platt Scaling, and that's probably a good choice. As @Ben Reinger noted, your calibrator should look like a "leftmost part of a sigmoid function" and you should be able to train such a calibrator with the sigmoid method. You can check that by following a tutorial on Beta Calibration. The author plots three different calibration models to asses their performance.

Note: the default binning strategy you're using (uniform) is probably a good choice despite the top answer. "strategy"="uniform" bins the predictions by regular predicted_score intervals. For example, bin [0, 0.1) contains all the predictions with a score in that range. However, there might be more predictions in the [0, 0.1) than in the [0.1, 0.2) bin. This is desirable since we can independently asses the calibration in disjoint bins. The alternative "strategy"="quantile" works by binning the predictions so that all contain the same number of samples. This doesn't work well with imbalanced data because the calibration curve will depend greatly on the huge number of low-score predictions (see the histogram of predicted scores). However, we are much more interested in the high-score bins (provided there is enough data in them).