I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated. However, I'm getting a very puzzling result:

xgb_clf = xgb.XGBClassifier(n_estimators=1000, 
                            scale_pos_weight = 10)

y_score_xgb = cross_val_predict(estimator=xgb_clf, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb[:,1], n_bins=10)

enter image description here

It seems like a "nice" (linear) reliability curve, however, the slope is less than 45 degrees.

and here is the classification report: enter image description here

However, if I do calibration, the resulting curve looks even worse:

calibrated = CalibratedClassifierCV(xgb_clf, method='sigmoid', cv=5)

y_score_xgb_clb = cross_val_predict(estimator=calibrated, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb_clb[:,1], n_bins=10)

enter image description here

What is more strange is that the outputted probabilities now clipped at ~0.75 (I don't get scores higher than 0.75).

Any suggestions / flaws in my approach?

  • 2
    $\begingroup$ there's a good chance your model is poorly calibrated because you set scale_pos_weight = 10. Try re-running the model with scale_pos_weight = 1. $\endgroup$ – Zach Sep 27 '19 at 15:28
  • 1
    $\begingroup$ I suspect your learning rate is too low vs number of trees. Has the error converged after 1000 trees? $\endgroup$ – seanv507 Jul 19 '20 at 15:43
  • $\begingroup$ how could scale pos weight be affecting this? if you have class imbalance isn't this parameter needed? if it is important to need well calibrated probabilities i would suggest optimizing brier score $\endgroup$ – Maths12 Aug 19 '20 at 12:41

I'm not sure "the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated" is correct: gradient boosting tends to push probability toward 0 and 1. Furthermore, you're applying weights, which should also skew your probabilities.

Because gradient boosting pushes probabilities outward rather than inward, using Platt scaling (method='sigmoid') is generally not the best bet. On the other hand, your original calibration plot does look vaguely like the leftmost part of a sigmoid function. But that explains why your recalibrated scores get cut off at 0.75: fitting a sigmoid onto your calibration plot (which isn't actually what happens, but close enough) will have the right half of the sigmoid cut off.

For expediency, I would first try method='isotonic'. For better understanding, I would suggest shifting scores to account for the weighting you gave, and see where the calibration plot sits then. (The shifting correction is better documented for logistic regression, but see Does down-sampling change logistic regression coefficients? and Convert predicted probabilities after downsampling to actual probabilities in classification .

Finally, sklearn's calibration_curve uses equal-width bins by default, which in an inbalanced dataset is probably not best. You might want to modify it to use equal-size (as in, number of datapoints) bins instead to get a better picture. In particular, I suspect the last two points on your second calibration curve represent very few datapoints, and should be taken with a grain of salt. (In sklearn v0.21, this became easier with the new parameter strategy='quantile'.)

  • $\begingroup$ It was my understanding that scale_pos_weight was used to weight gradient calculations, but not for the evaluation. It would make it different than plain oversampling. Any tought on that ? $\endgroup$ – lcrmorin Nov 7 '19 at 9:54
  • $\begingroup$ @lcrmorin, the gradient goes into the leaf scores: see eq5 in the paper (arxiv.org/pdf/1603.02754.pdf). It might help to think about the case without L2-regularization lambda=0, and loss=squared-loss so that h is constant. Then w^* is just the weighted average of the residuals in the leaf. See also stats.stackexchange.com/q/326110/232706 $\endgroup$ – Ben Reiniger Nov 8 '19 at 22:06
  • $\begingroup$ i am not understanding. if you have an imbalanced problem you are saying that we should use quantile binning? why? $\endgroup$ – Maths12 Aug 19 '20 at 12:36
  • 1
    $\begingroup$ @Maths12, for example in the post-calibration plot in OP, probably those two rightmost points consist of very few data points, and so are not very reliable. Adding some sort of error bar would be nice, but just ensuring that each plotted point summarizes a sufficient volume of the original data is helpful. $\endgroup$ – Ben Reiniger Aug 19 '20 at 14:01
  • $\begingroup$ so scale pos weight affects the weight formula (forumla 5 from the paper). If i am understanding this correctly then having a large scale pos weight will increase the weightings vector. This weights vector is given by eq 5 and is -ve. Therefore when it comes to calculating probabilities which is 1/(1+e^(-Xw)) , a high weighting vector will decrease the probability? since w is -ve $\endgroup$ – Maths12 Sep 1 '20 at 16:09

I'm not that familiar with gradient boosting, but I would assume that if you scale your minority class then your model will not be well calibrated. At the end of the day, it has learnt the distribution of the training data which does not reflect reality.

As for CalibratedClassifierCV, from reading the docs it seems that the sigmoid method is not applicable here given your distortion is not sigmoid shaped. Hence, if you have enough data that overfitting is not an issue, then why not try method='isotonic'?

  • $\begingroup$ why is it that if you scale your minority class then model will not be well calibrated? $\endgroup$ – Maths12 Aug 26 '20 at 15:25
  • $\begingroup$ A lot of times people do it unnecessarily especially with deep learning. But, if you have a small minority class with decision trees It can cause issues, you need enough examples of each class at each split for it to work effectively. $\endgroup$ – Anonymous Boar Aug 27 '20 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.