How to calibrate an XGBoost classifier which has been trained on a sampled dataset?

Question

I have trained my xgboost binary classifier on a dataset which does not represent the true proportion of positive over negative observations of the population. The model has approximately 45% of positives whereas the "true" population only has 15%.

Since I have to make inference on a sample which contains the original proportion of positives/negatives (15/85), how can I end up having calibrated probabilities?

I was thinking of reducing the positive observations in my training set after the xgboost training, so to restore the original proportions, and fit an isotonic regression on it. Then, I would use this model to calibrate my output probabilities:

from sklearn.isotonic import IsotonicRegression y_pred_train = (sampled y_pred_train) calibr = IsotonicRegression() calibr.fit(y_pred_train,y_train) y_pred_test = calibr.predict(y_pred_test)

Does this make any sense? Thanks in advance.

Answer 1

The calibration using isotonic regression will do two things for you, firstly re-adjusting for the oversampling, but secondly also helping with mis-calibration. The second point is rather helpful, because it is reasonably well-known that even if you had not oversampled, the calibration of XGBoost is often not right in the sense that on average cases predicted to be a 1 with probability X% do not end up being cases about X% of the time.

If it were not for the second issue, you could simply re-scale the log-odds by taking the inverse of he sigmoid function (i.e. log(predicted probability)- log(1 - predicted probability)) and adding a constant. I believe the constant would simply be the difference between 0.15 and 0.45 when transformed to the logit scale, i.e. -0.2006707 and -1.734601. I.e. you would get $$\text{adjusted prediction} = \sigma(\sigma^{-1}(\text{predicted probability})-1.53393).$$ In Python you'd probably use the logit and expit functions from scipy.special. However, that approach is only the way to do it, if your model is already well-calibrated, so what you propose seems better to me.