I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated. However, I'm getting a very puzzling result:

xgb_clf = xgb.XGBClassifier(n_estimators=1000, 
                            scale_pos_weight = 10)

y_score_xgb = cross_val_predict(estimator=xgb_clf, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb[:,1], n_bins=10)

enter image description here

It seems like a "nice" (linear) reliability curve, however, the slope is less than 45 degrees.

and here is the classification report: enter image description here

However, if I do calibration, the resulting curve looks even worse:

calibrated = CalibratedClassifierCV(xgb_clf, method='sigmoid', cv=5)

y_score_xgb_clb = cross_val_predict(estimator=calibrated, X=X, y=y, method='predict_proba', cv=5)

plot_calibration_curves(y_true=y, y_prob=y_score_xgb_clb[:,1], n_bins=10)

enter image description here

What is more strange is that the outputted probabilities now clipped at ~0.75 (I don't get scores higher than 0.75).

Any suggestions / flaws in my approach?

  • 1
    $\begingroup$ there's a good chance your model is poorly calibrated because you set scale_pos_weight = 10. Try re-running the model with scale_pos_weight = 1. $\endgroup$ – Zach Sep 27 '19 at 15:28

I'm not sure "the objective function of XGBoost is 'binary:logistic', the probabilities should be well calibrated" is correct: gradient boosting tends to push probability toward 0 and 1. Furthermore, you're applying weights, which should also skew your probabilities.

Because gradient boosting pushes probabilities outward rather than inward, using Platt scaling (method='sigmoid') is generally not the best bet. On the other hand, your original calibration plot does look vaguely like the leftmost part of a sigmoid function. But that explains why your recalibrated scores get cut off at 0.75: fitting a sigmoid onto your calibration plot (which isn't actually what happens, but close enough) will have the right half of the sigmoid cut off.

For expediency, I would first try method='isotonic'. For better understanding, I would suggest shifting scores to account for the weighting you gave, and see where the calibration plot sits then. (The shifting correction is better documented for logistic regression, but see Does down-sampling change logistic regression coefficients? and Convert predicted probabilities after downsampling to actual probabilities in classification .

Finally, sklearn's calibration_curve uses equal-width bins, which in an inbalanced dataset is probably not best. You might want to modify it to use equal-size (as in, number of datapoints) bins instead to get a better picture. In particular, I suspect the last two points on your second calibration curve represent very few datapoints, and should be taken with a grain of salt.

  • $\begingroup$ It was my understanding that scale_pos_weight was used to weight gradient calculations, but not for the evaluation. It would make it different than plain oversampling. Any tought on that ? $\endgroup$ – lcrmorin Nov 7 '19 at 9:54
  • $\begingroup$ @lcrmorin, the gradient goes into the leaf scores: see eq5 in the paper (arxiv.org/pdf/1603.02754.pdf). It might help to think about the case without L2-regularization lambda=0, and loss=squared-loss so that h is constant. Then w^* is just the weighted average of the residuals in the leaf. See also stats.stackexchange.com/q/326110/232706 $\endgroup$ – Ben Reiniger Nov 8 '19 at 22:06

I'm not that familiar with gradient boosting, but I would assume that if you scale your minority class then your model will not be well calibrated. At the end of the day, it has learnt the distribution of the training data which does not reflect reality.

As for CalibratedClassifierCV, from reading the docs it seems that the sigmoid method is not applicable here given your distortion is not sigmoid shaped. Hence, if you have enough data that overfitting is not an issue, then why not try method='isotonic'?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.