# LightGBM model improvement when the focus is on probability prediction

I am building a binary classifier using LightGBM. The goal is not to predict the outcome as such, but rather to predict the probability of the target even. To be more specific, it's more about ranking different objects based on the probability of the target event for them.

The dataset is imbalanced in that the distribution of classes is roughly 1 to 10. Not that the data is severely imbalanced, but this is definitely something that has its impact on the model's performance.

Given that probabilities are key for this task, I assumed that targeting the AUC score is more beneficial here especially given that it's somewhat immune to uneven class distributions.

I have a feeling that I didn't do a great job in feature engineering (I realize the importance of this part here), but let's assume for a moment that this is the dataset that I need to work with and all the feature engineering tricks have already been implemented.

Honestly speaking, I take it for granted that boosting-based models do not require much data wrangling. For instance, label encoding is enough and computationally expensive one-hot encoding can even be outperformed, etc.

With all that said, the results I get are far from perfect. Having an AUC score of 0.82 makes me think that in terms of probability prediction, the model is not awful, but the other metrics, as you can see, are satisfactory at best.

F1-score: 0.508
ROC AUC Score: 0.817
Cohen Kappa Score: 0.356


Analyzing the precision/recall curve and trying to find the threshold that sets their ratio to $$\approx1$$ yields a more balanced situation, but for this task, it's not yet clear which type of error should be minimized or whether, say, f1-score maximization is the target.

Anyway, all the conventional metrics are dependent upon the chosen threshold so it's not clear whether I can just save time on threshold tuning.

My questions:

1. Would it be correct to state that having a reasonably high AUC for such tasks can be prioritized as opposed to just looking at precision, recall and other metrics that are functions of thresholds?

2. I use a combination of Optuna and 5-fold cross-validation to select the best hyperparameters. The results, however, do not improve significantly. I cannot even get a very high AUC score on the train dataset regardless of the number of estimators used for LGBMClassifier. Does it mean that this is some kind of plateau for this task, dataset, and features?
What are some common methods (in addition to better feature engineering and getting more data) to improve gradient boosting methods' results?

             precision    recall  f1-score   support

False       0.92      0.76      0.83     10902
True       0.40      0.70      0.51      2482

accuracy                           0.75     13384
macro avg       0.66      0.73      0.67     13384
weighted avg       0.82      0.75      0.77     13384

Results for threshold=0.66:
precision    recall  f1-score   support

False       0.89      0.89      0.89     10902
True       0.52      0.51      0.51      2482

accuracy                           0.82     13384
macro avg       0.70      0.70      0.70     13384
weighted avg       0.82      0.82      0.82     13384

F1-score: 0.515
ROC AUC Score: 0.817
Cohen Kappa Score: 0.405
$$$$
`
• Optimizing the default metric (log-loss) is usually not the worst thing to do. It is the same metric that is optimized by logistic regression and corresponds to the usual objective function of LGB for binary target. To get a feeling of your model performance, you can calculate ROC AUC as well. Mar 31, 2021 at 18:20
• Using the binary log loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1 for this task but not our best bet necessarily. For the Qs: 1.Yes, but we can potentially do better (see prior comment). 2. Try different hyper-parameters as well as learners in general; LightGBM is awesome but not a panacea. Apr 1, 2021 at 11:30
• @usεr11852, thank you for your comment. Let me ask you also whether it's necessary to do any kind of calibration (Platt or isotonic) after training. I know that it's required when you try to oversample/undersample your data, but is it true in general for boosted trees when you don't resample your data? Apr 1, 2021 at 12:42
• @usεr11852 why didn't you write a complete answer? you gave it all. Apr 11, 2021 at 9:19
• @carlo: Thank you for the vote of confidence. :D Apr 12, 2021 at 23:33