2
$\begingroup$

Why does my XGBClassifier predicts probability only from 0.48 to 0.51 for either class?

I'm very new to XGBoost, so any suggestions are greatly appreciated! Here's what I want to do using python:

  • I have a binary classification problem
  • I want to get predicted probability for thresholding (so I want predict_proba())
  • Based on what I've read, XGBClassifier supports predict_proba(), so that's what I'm using

However, after I trained the model (hyperparameters at the end of the post), when I use model.predict_proba(val_X), the output only ranges from 0.48 to 0.51 for either class. Something like this:

enter image description here

I've read a few other posts like "xgboost logistic regression predictions are returning values >1 and < 0", and "How does gradient boosting calculate probability estimates?", but I can't figure out if the predict_proba() is giving me log odds or predicted probability, or what exactly is happening.

An answer to this post "Unexpected probability distribution from xgboost binary classification" suggests that the model may not be learning anything from the data, and therefore the random probabilities. My roc-auc is 0.7662914691943129 and although I have class imbalance issue (80% of positive class), I've used the balanced class weights (n_samples / (n_classes * np.bincount(y))) during model training.

Here are the parameters for my model (I've used optuna to tune my hyperparameters):

 {'objective': 'binary:logistic',
 'use_label_encoder': False,
 'base_score': 0.5,
 'booster': 'gbtree',
 'colsample_bylevel': 1,
 'colsample_bynode': 1,
 'colsample_bytree': 1,
 'enable_categorical': False,
 'gamma': 0.20869504071834755,
 'gpu_id': -1,
 'importance_type': None,
 'interaction_constraints': '',
 'learning_rate': 9.48345478e-05,
 'max_delta_step': 0,
 'max_depth': 9,
 'min_child_weight': 1,
 'missing': nan,
 'monotone_constraints': '()',
 'n_estimators': 15,
 'n_jobs': 8,
 'num_parallel_tree': 1,
 'predictor': 'auto',
 'random_state': 0,
 'reg_alpha': 0.980300725,
 'reg_lambda': 0.00221248553,
 'scale_pos_weight': 0.24369747899159663,
 'subsample': 1,
 'tree_method': 'exact',
 'validate_parameters': 1,
 'verbosity': 0,
 'eval_metric': ['auc', 'logloss'],
 'lambda': 0.002212485584996869,
 'alpha': 0.980300751529644,
 'eta': 9.483455063850674e-05,
 'grow_policy': 'depthwise'}
$\endgroup$
3
  • $\begingroup$ What's your question? Please edit to clarify. $\endgroup$
    – Sycorax
    Commented Feb 17, 2022 at 22:59
  • 3
    $\begingroup$ Did your model perhaps fit only a small number of trees? A constant (or close to constant) prediction is correct -- it's the optimal thing to do -- in problems where X is not very predictive of Y. If you don't have much signal, your predictions will be close to constant. $\endgroup$
    – Adrian
    Commented Feb 17, 2022 at 23:10
  • $\begingroup$ Thanks Adrian! I think that's what's happening! I may have too much regularization and it prevented the model from learning the signal. $\endgroup$
    – yuan-ning
    Commented Feb 18, 2022 at 20:27

1 Answer 1

3
$\begingroup$

As @Adrian already pointed out, the reason the predicted probabilities are close to 0.5 is because the model has very small number of trees

'n_estimators': 15 

and at the same time a small learning rate

'learning_rate': 9.48345478e-05

which cannot produce predictions that deviate much from the initial prediction of 0.5 by default.

One further comment is that these results from hyperparameter search is likely due to using roc-auc as (stopping) criteria. These predicted probabilities may be sufficient to produce optimal rank ordering metrics, but not necessarily producing probability close to expected values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.