XGBoost Calibration for weighted loss function

Question

I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is multi:softprob, so that I can get predicted probabilities for each class. I also have a vector of weights for each of the observations I'm using. I've plotted a calibration curve for each class (basically using a One vs. Rest approach, as that is what's done by CalibratedClassifierCV for multiclass classification). As expected by the nature of XGBoost, the distribution always looks bimodal with modes very close to 0 and 1.

I would like to calibrate my model but I am unsure how this can be done (preferably in R). I don't mind using a One vs. Rest approach as I understand that's what's done in Python and it's probably the simplest approach out there. My question is how this can be done given that the calibration process is apparently using the Brier score as objective. Does my choice of loss function affect things and how can I account for my observation weights in the calibration process? Any suggestions would be very helpful!

Answer 1

If you are doing the one vs. rest approach you are essentially doing calibration for a binary problem. The idea behind this is to get predictions which are as close as possible to the conditional event probabilities and this is independent of the loss function you use.

Mathematically, if $Y$ is your observation and $X$ is your predicted probability, you want $\mathbb{P}(Y=1 | X = p) \approx p$ to hold. So you basically have to estimate the probability of an event given that your predicted probability falls into a certain region. This can for instance be achieved with some sort of regression where $Y$ is the dependent variable.

For more details you can read this paper on Stable reliability diagrams for probabilistic classifiers which also provides some R code for reliability diagrams and recalibration based on isotonic regression.