After having read a few articles and papers on probability calibration, I still don't have a clear understanding of what could be the best way to do model calibration in my case.
I am using LightGBM (gradient boosting library) to do binary classification. The distribution of classes is roughly 1:5 so the dataset is imbalanced but it's not that bad.
As always, it's very important to understand the application of the model first. In my case, the focus is on the model's ability to rank objects based on their propensity to commit the target event.
My understanding was that it's not too wise to make decisions based on threshold-dependent metrics like f1-score and I should prioritize AUC score. This metric is known to reflect the ranking capability of the model better and is generally immune to imbalanced datasets (correct me if I am wrong about anything).
Regardless of the hyperparameter tuning, I get an AUC score of 0.8, which is not bad, but the threshold-based metrics (precision, recall, and hence f1-score) for the positive class are at best around 0.48-0.5.
What is also not great is that the outputs of the model are concentrated around 0.5-0.6 which doesn't make a lot of sense so I started thinking about model calibration.
As far as I understand, there are a lot of options, but I would like to consider the simplest first:
- Platt (sigmoid) Calibration
- Isotonic regression
- Using focal loss instead of logloss
I have also read a few times that we can optimize the Brier loss and thus get better calibration. Is that correct?
A few more questions:
Would it be correct to use focal or Brier loss and then do additional calibration using Platt/isotonic regression?
Given what I described above, is it possible to answer confidently whether Platt/isotonic, Brier/focal is better, or is it all just trial and error?
Are there any additional ways you can think of to improve the classification results? Let's assume for a while that I have all the features available and nothing else can be improved from the feature engineering perspective.
In general, does it make sense to have another holdout dataset exclusively for calibration or we can do that in parallel with model fitting? I have seen some arguments regarding probability calibration on the test dataset but that sounds like overfitting to me.