I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated. However, I'm getting a very puzzling result:

xgb_clf = xgb.XGBClassifier(n_estimators=1000, 
                            scale_pos_weight = 10)

y_score_xgb = cross_val_predict(estimator=xgb_clf, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb[:,1], n_bins=10)

enter image description here

It seems like a "nice" (linear) reliability curve, however, the slope is less than 45 degrees.

and here is the classification report: enter image description here

However, if I do calibration, the resulting curve looks even worse:

calibrated = CalibratedClassifierCV(xgb_clf, method='sigmoid', cv=5)

y_score_xgb_clb = cross_val_predict(estimator=calibrated, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb_clb[:,1], n_bins=10)

enter image description here

What is more strange is that the outputted probabilities now clipped at ~0.75 (I don't get scores higher than 0.75).

Any suggestions / flaws in my approach?

  • 5
    $\begingroup$ there's a good chance your model is poorly calibrated because you set scale_pos_weight = 10. Try re-running the model with scale_pos_weight = 1. $\endgroup$
    – Zach
    Commented Sep 27, 2019 at 15:28
  • 1
    $\begingroup$ I suspect your learning rate is too low vs number of trees. Has the error converged after 1000 trees? $\endgroup$
    – seanv507
    Commented Jul 19, 2020 at 15:43
  • 1
    $\begingroup$ how could scale pos weight be affecting this? if you have class imbalance isn't this parameter needed? if it is important to need well calibrated probabilities i would suggest optimizing brier score $\endgroup$
    – Maths12
    Commented Aug 19, 2020 at 12:41

3 Answers 3


I'm not sure "the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated" is correct: gradient boosting tends to push probability toward 0 and 1. More significantly, you're applying weights (scale_pos_weight=10), which will skew your probabilities higher than the data would suggest.

Because gradient boosting pushes probabilities outward rather than inward, using Platt scaling (method='sigmoid') is generally not the best bet. On the other hand, your original calibration plot does look vaguely like the leftmost part of a sigmoid function. But that explains why your recalibrated scores get cut off at 0.75: fitting a sigmoid onto your calibration plot (which isn't actually what happens, but close enough) will have the right half of the sigmoid cut off.

For expediency, I would first try method='isotonic'. For better understanding, I would suggest shifting scores to account for the weighting you gave, and see where the calibration plot sits then. (The shifting correction is better documented for logistic regression, but see Does down-sampling change logistic regression coefficients? and Convert predicted probabilities after downsampling to actual probabilities in classification .

Finally, sklearn's calibration_curve uses equal-width bins by default, which in an inbalanced dataset is probably not best. You might want to modify it to use equal-size (as in, number of datapoints) bins instead to get a better picture. In particular, I suspect the last two points on your second calibration curve represent very few datapoints, and should be taken with a grain of salt. (In sklearn v0.21, this became easier with the new parameter strategy='quantile'.)

  • $\begingroup$ It was my understanding that scale_pos_weight was used to weight gradient calculations, but not for the evaluation. It would make it different than plain oversampling. Any tought on that ? $\endgroup$ Commented Nov 7, 2019 at 9:54
  • $\begingroup$ @lcrmorin, the gradient goes into the leaf scores: see eq5 in the paper (arxiv.org/pdf/1603.02754.pdf). It might help to think about the case without L2-regularization lambda=0, and loss=squared-loss so that h is constant. Then w^* is just the weighted average of the residuals in the leaf. See also stats.stackexchange.com/q/326110/232706 $\endgroup$ Commented Nov 8, 2019 at 22:06
  • $\begingroup$ i am not understanding. if you have an imbalanced problem you are saying that we should use quantile binning? why? $\endgroup$
    – Maths12
    Commented Aug 19, 2020 at 12:36
  • 1
    $\begingroup$ @Maths12, for example in the post-calibration plot in OP, probably those two rightmost points consist of very few data points, and so are not very reliable. Adding some sort of error bar would be nice, but just ensuring that each plotted point summarizes a sufficient volume of the original data is helpful. $\endgroup$ Commented Aug 19, 2020 at 14:01
  • $\begingroup$ so scale pos weight affects the weight formula (forumla 5 from the paper). If i am understanding this correctly then having a large scale pos weight will increase the weightings vector. This weights vector is given by eq 5 and is -ve. Therefore when it comes to calculating probabilities which is 1/(1+e^(-Xw)) , a high weighting vector will decrease the probability? since w is -ve $\endgroup$
    – Maths12
    Commented Sep 1, 2020 at 16:09

I'm not that familiar with gradient boosting, but I would assume that if you scale your minority class then your model will not be well calibrated. At the end of the day, it has learnt the distribution of the training data which does not reflect reality.

As for CalibratedClassifierCV, from reading the docs it seems that the sigmoid method is not applicable here given your distortion is not sigmoid shaped. Hence, if you have enough data that overfitting is not an issue, then why not try method='isotonic'?

  • $\begingroup$ why is it that if you scale your minority class then model will not be well calibrated? $\endgroup$
    – Maths12
    Commented Aug 26, 2020 at 15:25
  • $\begingroup$ A lot of times people do it unnecessarily especially with deep learning. But, if you have a small minority class with decision trees It can cause issues, you need enough examples of each class at each split for it to work effectively. $\endgroup$
    – Anonymous
    Commented Aug 27, 2020 at 17:14


No wonder your post-calibration plot looks like this. Look at your reliability curve pre-calibration. The more predicted score grows, the more actual positives it picks up. However, the best your model can do is to extract around 20% of actual positives (when the predicted score is over 0.9). The ideal calibrator would squeeze your probability predictions into [0, 0.2] interval because your model can't do any better. In other words, a great calibrator would map your orange points onto the diagonal line by moving them approximately sideways to the left. You can't expect to get a well-calibrated score of 0.7 ever with this model because it's just not as good.

Additional remarks:

You're using Platt Scaling, and that's probably a good choice. As @Ben Reinger noted, your calibrator should look like a "leftmost part of a sigmoid function" and you should be able to train such a calibrator with the sigmoid method. You can check that by following a tutorial on Beta Calibration. The author plots three different calibration models to asses their performance.

Note: the default binning strategy you're using (uniform) is probably a good choice despite the top answer. "strategy"="uniform" bins the predictions by regular predicted_score intervals. For example, bin [0, 0.1) contains all the predictions with a score in that range. However, there might be more predictions in the [0, 0.1) than in the [0.1, 0.2) bin. This is desirable since we can independently asses the calibration in disjoint bins. The alternative "strategy"="quantile" works by binning the predictions so that all contain the same number of samples. This doesn't work well with imbalanced data because the calibration curve will depend greatly on the huge number of low-score predictions (see the histogram of predicted scores). However, we are much more interested in the high-score bins (provided there is enough data in them).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.