4
$\begingroup$

I am building a binary classifier using LightGBM. The goal is not to predict the outcome as such, but rather to predict the probability of the target even. To be more specific, it's more about ranking different objects based on the probability of the target event for them.

The dataset is imbalanced in that the distribution of classes is roughly 1 to 10. Not that the data is severely imbalanced, but this is definitely something that has its impact on the model's performance.

Given that probabilities are key for this task, I assumed that targeting the AUC score is more beneficial here especially given that it's somewhat immune to uneven class distributions.

I have a feeling that I didn't do a great job in feature engineering (I realize the importance of this part here), but let's assume for a moment that this is the dataset that I need to work with and all the feature engineering tricks have already been implemented.

Honestly speaking, I take it for granted that boosting-based models do not require much data wrangling. For instance, label encoding is enough and computationally expensive one-hot encoding can even be outperformed, etc.

With all that said, the results I get are far from perfect. Having an AUC score of 0.82 makes me think that in terms of probability prediction, the model is not awful, but the other metrics, as you can see, are satisfactory at best.

F1-score: 0.508
ROC AUC Score: 0.817
Cohen Kappa Score: 0.356

Analyzing the precision/recall curve and trying to find the threshold that sets their ratio to $\approx1$ yields a more balanced situation, but for this task, it's not yet clear which type of error should be minimized or whether, say, f1-score maximization is the target.

Anyway, all the conventional metrics are dependent upon the chosen threshold so it's not clear whether I can just save time on threshold tuning.

My questions:

  1. Would it be correct to state that having a reasonably high AUC for such tasks can be prioritized as opposed to just looking at precision, recall and other metrics that are functions of thresholds?

  2. I use a combination of Optuna and 5-fold cross-validation to select the best hyperparameters. The results, however, do not improve significantly. I cannot even get a very high AUC score on the train dataset regardless of the number of estimators used for LGBMClassifier. Does it mean that this is some kind of plateau for this task, dataset, and features?
    What are some common methods (in addition to better feature engineering and getting more data) to improve gradient boosting methods' results?

             precision    recall  f1-score   support

       False       0.92      0.76      0.83     10902
        True       0.40      0.70      0.51      2482

    accuracy                           0.75     13384
   macro avg       0.66      0.73      0.67     13384
weighted avg       0.82      0.75      0.77     13384
Results for threshold=0.66:
              precision    recall  f1-score   support

       False       0.89      0.89      0.89     10902
        True       0.52      0.51      0.51      2482

    accuracy                           0.82     13384
   macro avg       0.70      0.70      0.70     13384
weighted avg       0.82      0.82      0.82     13384

F1-score: 0.515
ROC AUC Score: 0.817
Cohen Kappa Score: 0.405
```
$\endgroup$
6
  • $\begingroup$ Optimizing the default metric (log-loss) is usually not the worst thing to do. It is the same metric that is optimized by logistic regression and corresponds to the usual objective function of LGB for binary target. To get a feeling of your model performance, you can calculate ROC AUC as well. $\endgroup$ – Michael M Mar 31 at 18:20
  • $\begingroup$ Using the binary log loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1 for this task but not our best bet necessarily. For the Qs: 1.Yes, but we can potentially do better (see prior comment). 2. Try different hyper-parameters as well as learners in general; LightGBM is awesome but not a panacea. $\endgroup$ – usεr11852 Apr 1 at 11:30
  • $\begingroup$ @usεr11852, thank you for your comment. Let me ask you also whether it's necessary to do any kind of calibration (Platt or isotonic) after training. I know that it's required when you try to oversample/undersample your data, but is it true in general for boosted trees when you don't resample your data? $\endgroup$ – Don Draper Apr 1 at 12:42
  • 2
    $\begingroup$ @usεr11852 why didn't you write a complete answer? you gave it all. $\endgroup$ – carlo Apr 11 at 9:19
  • 2
    $\begingroup$ @carlo: Thank you for the vote of confidence. :D $\endgroup$ – usεr11852 Apr 12 at 23:33
4
+50
$\begingroup$

Using the binary log-loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna (or our general hyper-parameter search framework) to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1-score for this task but not our best bet necessarily.

Regarding the particular questions in the main post:

  1. Yes, but we can potentially do better (as discussed above). Using metrics based on discontinuous rules like Precision, Recall, F1, etc. can be misleading. This post on Is accuracy an improper scoring rule in a binary classification setting? focuses on Accuracy but the same applies for metrics like Precision, etc.
  2. Try different hyper-parameters as well as learners; LightGBM is awesome but not a panacea. Even simply trying XGBoost and Catboost might be enough to explore some obvious easy pickings.

Regarding the sub-question in the comments:

  1. Using isotonic regression can be beneficial but it has to be setup carefully (hold-out sets, etc.). I do it irrespective of "resampling" if I have time but usually it give me little gains in terms of ROC-/PR-AUC. It might worth considering other calibration options too like Platt scaling and beta calibration; I have not found one to dominate over the others in my work though.
  2. Please see my answer in the CV.SE thread: Biased prediction (overestimation) for xgboost I think it is pertinent to your question. As mentioned there, (early) gradient boosting implementation are (were?) not very well calibrated. With larger datasets and more well-designed loss-functions this might have been ameliorated nowadays to some extent but I have not seen any recent papers.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.