1
$\begingroup$

After having read a few articles and papers on probability calibration, I still don't have a clear understanding of what could be the best way to do model calibration in my case.

I am using LightGBM (gradient boosting library) to do binary classification. The distribution of classes is roughly 1:5 so the dataset is imbalanced but it's not that bad.

As always, it's very important to understand the application of the model first. In my case, the focus is on the model's ability to rank objects based on their propensity to commit the target event.

My understanding was that it's not too wise to make decisions based on threshold-dependent metrics like f1-score and I should prioritize AUC score. This metric is known to reflect the ranking capability of the model better and is generally immune to imbalanced datasets (correct me if I am wrong about anything).

Regardless of the hyperparameter tuning, I get an AUC score of 0.8, which is not bad, but the threshold-based metrics (precision, recall, and hence f1-score) for the positive class are at best around 0.48-0.5.

What is also not great is that the outputs of the model are concentrated around 0.5-0.6 which doesn't make a lot of sense so I started thinking about model calibration.

As far as I understand, there are a lot of options, but I would like to consider the simplest first:

  • Platt (sigmoid) Calibration
  • Isotonic regression
  • Using focal loss instead of logloss

I have also read a few times that we can optimize the Brier loss and thus get better calibration. Is that correct?

A few more questions:

  1. Would it be correct to use focal or Brier loss and then do additional calibration using Platt/isotonic regression?

  2. Given what I described above, is it possible to answer confidently whether Platt/isotonic, Brier/focal is better, or is it all just trial and error?

  3. Are there any additional ways you can think of to improve the classification results? Let's assume for a while that I have all the features available and nothing else can be improved from the feature engineering perspective.

  4. In general, does it make sense to have another holdout dataset exclusively for calibration or we can do that in parallel with model fitting? I have seen some arguments regarding probability calibration on the test dataset but that sounds like overfitting to me.

Thanks!

$\endgroup$
2
  • 1
    $\begingroup$ You should be able to rig the recall to be perfect by setting an extreme threshold and classifying everything the same way, so I do not follow why yours is stuck at $0.5$. $\endgroup$ – Dave Apr 12 at 10:21
  • $\begingroup$ Setting an extreme threshold can help with this trade-off so it's possible, as you mentioned, to get a decent recall. I should have articulated more clearly that I was trying to balance precision and recall using the precision/recall curve and the "balanced" value is around 0.5 $\endgroup$ – Don Draper Apr 12 at 10:29
1
+50
$\begingroup$

ROC-AUC always tracks with precision, recall, and f1-score, and since you are obtaining AUC values of 0.8 with everything else near 0.48-0.5, you are obviously overfitting. You didn't say anything about number of objects, features, folds, or anything about how you are testing objects left out of training. It seems that you are also employing one data set, one feature set, and one classifier, so it's hard to believe your results in terms of being able to overcome the bias-variance dilemma. In addition, how can you guarantee the robustness of your current model in different environments based on different data sets and different feature sets?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.