I am building a binary classifier using LightGBM. The goal is not to predict the outcome as such, but rather to predict the probability of the target even. To be more specific, it's more about ranking different objects based on the probability of the target event for them.

The dataset is imbalanced in that the distribution of classes is roughly 1 to 10. Not that the data is severely imbalanced, but this is definitely something that has its impact on the model's performance.

Given that probabilities are key for this task, I assumed that targeting the AUC score is more beneficial here especially given that it's somewhat immune to uneven class distributions.

I have a feeling that I didn't do a great job in feature engineering (I realize the importance of this part here), but let's assume for a moment that this is the dataset that I need to work with and all the feature engineering tricks have already been implemented.

Honestly speaking, I take it for granted that boosting-based models do not require much data wrangling. For instance, label encoding is enough and computationally expensive one-hot encoding can even be outperformed, etc.

With all that said, the results I get are far from perfect. Having an AUC score of 0.82 makes me think that in terms of probability prediction, the model is not awful, but the other metrics, as you can see, are satisfactory at best.

F1-score: 0.508
ROC AUC Score: 0.817
Cohen Kappa Score: 0.356

Analyzing the precision/recall curve and trying to find the threshold that sets their ratio to $\approx1$ yields a more balanced situation, but for this task, it's not yet clear which type of error should be minimized or whether, say, f1-score maximization is the target.

Anyway, all the conventional metrics are dependent upon the chosen threshold so it's not clear whether I can just save time on threshold tuning.

My questions:

  1. Would it be correct to state that having a reasonably high AUC for such tasks can be prioritized as opposed to just looking at precision, recall and other metrics that are functions of thresholds?

  2. I use a combination of Optuna and 5-fold cross-validation to select the best hyperparameters. The results, however, do not improve significantly. I cannot even get a very high AUC score on the train dataset regardless of the number of estimators used for LGBMClassifier. Does it mean that this is some kind of plateau for this task, dataset, and features?
    What are some common methods (in addition to better feature engineering and getting more data) to improve gradient boosting methods' results?

             precision    recall  f1-score   support

       False       0.92      0.76      0.83     10902
        True       0.40      0.70      0.51      2482

    accuracy                           0.75     13384
   macro avg       0.66      0.73      0.67     13384
weighted avg       0.82      0.75      0.77     13384
Results for threshold=0.66:
              precision    recall  f1-score   support

       False       0.89      0.89      0.89     10902
        True       0.52      0.51      0.51      2482

    accuracy                           0.82     13384
   macro avg       0.70      0.70      0.70     13384
weighted avg       0.82      0.82      0.82     13384

F1-score: 0.515
ROC AUC Score: 0.817
Cohen Kappa Score: 0.405
  • $\begingroup$ Optimizing the default metric (log-loss) is usually not the worst thing to do. It is the same metric that is optimized by logistic regression and corresponds to the usual objective function of LGB for binary target. To get a feeling of your model performance, you can calculate ROC AUC as well. $\endgroup$
    – Michael M
    Mar 31, 2021 at 18:20
  • $\begingroup$ Using the binary log loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1 for this task but not our best bet necessarily. For the Qs: 1.Yes, but we can potentially do better (see prior comment). 2. Try different hyper-parameters as well as learners in general; LightGBM is awesome but not a panacea. $\endgroup$
    – usεr11852
    Apr 1, 2021 at 11:30
  • $\begingroup$ @usεr11852, thank you for your comment. Let me ask you also whether it's necessary to do any kind of calibration (Platt or isotonic) after training. I know that it's required when you try to oversample/undersample your data, but is it true in general for boosted trees when you don't resample your data? $\endgroup$
    – Don Draper
    Apr 1, 2021 at 12:42
  • 2
    $\begingroup$ @usεr11852 why didn't you write a complete answer? you gave it all. $\endgroup$
    – carlo
    Apr 11, 2021 at 9:19
  • 2
    $\begingroup$ @carlo: Thank you for the vote of confidence. :D $\endgroup$
    – usεr11852
    Apr 12, 2021 at 23:33

1 Answer 1


Using the binary log-loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna (or our general hyper-parameter search framework) to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1-score for this task but not our best bet necessarily.

Regarding the particular questions in the main post:

  1. Yes, but we can potentially do better (as discussed above). Using metrics based on discontinuous rules like Precision, Recall, F1, etc. can be misleading. This post on Is accuracy an improper scoring rule in a binary classification setting? focuses on Accuracy but the same applies for metrics like Precision, etc.
  2. Try different hyper-parameters as well as learners; LightGBM is awesome but not a panacea. Even simply trying XGBoost and Catboost might be enough to explore some obvious easy pickings.

Regarding the sub-question in the comments:

  1. Using isotonic regression can be beneficial but it has to be setup carefully (hold-out sets, etc.). I do it irrespective of "resampling" if I have time but usually it give me little gains in terms of ROC-/PR-AUC. It might worth considering other calibration options too like Platt scaling and beta calibration; I have not found one to dominate over the others in my work though.
  2. Please see my answer in the CV.SE thread: Biased prediction (overestimation) for xgboost I think it is pertinent to your question. As mentioned there, (early) gradient boosting implementation are (were?) not very well calibrated. With larger datasets and more well-designed loss-functions this might have been ameliorated nowadays to some extent but I have not seen any recent papers.
  • 1
    $\begingroup$ How is AUC better than F1 for an imbalanced data set? I disagree as AUC can be misleading if the majority class is predicted correctly and the minority class isn't. F1 will provide better notice if the minority class isn't well predicted, as seen on this example with AUC 0.8 and F1 0.5 $\endgroup$
    – PJ_
    Sep 23, 2022 at 12:52
  • $\begingroup$ @PeJota: Especially when dealing with an imbalanced data we need to account for misclassification costs when assessing our model's usefulness. To do that label assignment we need to define "some threshold" - that is not bad or good, it is a necessity. What is potentially bad and misleading is using an arbitrary threshold (e.g. 0.50) to calculate model performance. When we are therefore in a model development stage it makes more sense to use metrics like ROC-AUC or PR-AUC that give us a more holistic view of a model's performance - this is a different from evaluating (cont.) $\endgroup$
    – usεr11852
    Sep 25, 2022 at 1:19
  • $\begingroup$ our model's utility (via F1 or whatever). Finally do note that a metric value on it's own ("F1: 0.50") means very little. Maybe it is a bad result, maybe it is a world-leading result - without a baseline/reference strategy it is almost impossible to tell. See for example a paper like Assel et al. (2017) The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models. They correctly raise a similar point arguing "against Brier score" supporting correctly a decision-analytic approach. $\endgroup$
    – usεr11852
    Sep 25, 2022 at 1:19
  • $\begingroup$ @PeJota true what you say about the AUC, but F1 score is a threshold metric and ultimately a binary metric, what one would want to use is probably in this case is Area Under Precision recall Curve. $\endgroup$ Sep 29, 2022 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.