Probability Calibration for Highly Imbalanced Binary Classification

Question

I am working on a binary classification problem on a highly imbalanced dataset (1:100) where model probabilities are important for the use case and need to be well calibrated to best represent true probabilities for the minority class. I have trained several models and am using class weight parameters during the model fitting process to account for class imbalance. The classifiers I have trained and the associated class weight parameters I am using for each are as follows: RandomForestClassifier(class_weight='balanced'), XGBClassifier(scale_pos_weight=100), LGBMClassifier(class_weight='balanced'), CatBoostClassifier(auto_class_weights='Balanced').

I have trained each of these models on train data before creating a CalibratedClassifierCV instance and specifying cv='prefit' to flag that each model has already been fit. I am then calling the fit method for each CalibratedClassifierCV instance on separate validation data to calibrate model probabilities using both isotonic and sigmoid calibration methods. Using sklearn's CalibrationDisplay I have created calibration curves and histogram plots binning mean model probability scores for each model on out-of-time data. Refer to the plots below:

From the plots above I have two primary concerns/questions:

1. Are calibration plots relevant for highly imbalanced datasets? Clearly none of the models evaluated are close to the hugging the diagonal representing a perfectly calibrated model, but is this more of a function of the severe class imbalance or poorly calibrated model probability scores?

2. In the histogram plots binning mean model probability scores, what explains the difference between the wide range of probability scores observed for uncalibrated RandomForest and CatBoost models as compared to the much smaller and range of low probability scores observed for uncalibrated XGBoost and LightGBM models? Intuitively, I would expect a well calibrated model for such a highly imbalanced dataset to show binned mean model probability scores as more of an extreme non-centered Poisson distribution as is observed in the plots for uncalibrated XGBoost and LightGBM as well as all other calibrated models. What explains the wide range of probability scores observed for uncalibrated RandomForest and CatBoost models?

Answer 1

Oversampling or weighting (which comes to the same thing) will explicitly bias your predictions towards the minority class, so it directly contradicts your goal of having well-calibrated probabilistic classifications. This should show up in overestimates of the predicted probabilities, i.e., lines below the diagonal in the calibration plot, which is exactly what we see.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? See also the links collected at Profusion of threads on imbalanced data - can we merge/deem canonical any?

Classification diagrams have the same meaning and relevance in "unbalanced" situation as in "balanced" ones. The problem of course is that with a low incidence of your target class, parameter estimates and predictions will come with high variance. And of course, even the conditional class membership probabilities will always be low (both actual and predicted), so the farther to the right and up you go in the diagram, the less reliable it is.

I am sorry not to have an answer to your second question.