I am working on a binary classification problem on a highly imbalanced dataset (1:100) where model probabilities are important for the use case and need to be well calibrated to best represent true probabilities for the minority class. I have trained several models and am using class weight parameters during the model fitting process to account for class imbalance. The classifiers I have trained and the associated class weight parameters I am using for each are as follows:
I have trained each of these models on train data before creating a
CalibratedClassifierCV instance and specifying
cv='prefit' to flag that each model has already been fit. I am then calling the fit method for each
CalibratedClassifierCV instance on separate validation data to calibrate model probabilities using both isotonic and sigmoid calibration methods. Using sklearn's
CalibrationDisplay I have created calibration curves and histogram plots binning mean model probability scores for each model on out-of-time data. Refer to the plots below:
From the plots above I have two primary concerns/questions:
Are calibration plots relevant for highly imbalanced datasets? Clearly none of the models evaluated are close to the hugging the diagonal representing a perfectly calibrated model, but is this more of a function of the severe class imbalance or poorly calibrated model probability scores?
In the histogram plots binning mean model probability scores, what explains the difference between the wide range of probability scores observed for uncalibrated RandomForest and CatBoost models as compared to the much smaller and range of low probability scores observed for uncalibrated XGBoost and LightGBM models? Intuitively, I would expect a well calibrated model for such a highly imbalanced dataset to show binned mean model probability scores as more of an extreme non-centered Poisson distribution as is observed in the plots for uncalibrated XGBoost and LightGBM as well as all other calibrated models. What explains the wide range of probability scores observed for uncalibrated RandomForest and CatBoost models?