9
$\begingroup$

I'm using XGBoost for a binary classification task—trying to predict whether team A will beat team B given the score of the game and the time left. I know for certain score-time combinations, the probability of a success (team A wins) is essentially 0 or 1 (e.g. up 20 points with a minute left, it should be ~0.999). This is true empirically (in previous games) regardless of the other features (team quality, pace of play, etc.)

The problem: When I look at fitted values given these score-time combos, the output tends to be around 0.03 or 0.97, rather than .0001 or .9999. I've tried loads of different combinations of parameters and can't get the model to output something close to 0 or 1.

A few more details:

  • I have about 630k observations, about 55% of which are successes. I also have at least tens of thousands where I believe the probability should be > .99 or < .01.
  • Parameters I've messed with + the ranges I've tried: max_depth (from 4 to 12), n_estimators (up to a couple hundred), eta (.001 to .3), min_child_weight (up to a couple hundred), reg_lambda (1-5), gamma (0-5), colsample_byX (0.7-1), and subsample (0.7-1)
  • I have not messed with scale_pos_weight since my understanding is that this parameter is for imbalanced datasets. My dataset isn't imbalanced overall, it's just imbalanced at certain places in the feature space
  • Loss function is binary:logistic

Any thoughts? Thanks a ton!

$\endgroup$
5
  • 2
    $\begingroup$ How do you know that the probability should be $10^{-4}$ instead of 0.03? What hyperparmeters are you using? How many observations, features, and members of each class do you have? What is your loss function? Please edit to clarify. $\endgroup$
    – Sycorax
    Mar 2, 2022 at 19:12
  • 3
    $\begingroup$ Are 0.03 and 0.97 meaningly different from .0001 or .9999? Are you still classifying 0.03 as 0 and 0.97 as 1? If so, it's probably not worth examining further. $\endgroup$
    – Eli
    Mar 2, 2022 at 19:13
  • 1
    $\begingroup$ @Sycorax edited. Thanks! One comment on features: Right now it's just time, score, and the teams' qualities. I'll add more eventually, but given these features alone, I would expect the model to pick up on the fact that given certain time-score combinations, there aren't any features that are going to drag the probability away from 0 or 1 $\endgroup$
    – dfried
    Mar 2, 2022 at 19:20
  • 1
    $\begingroup$ If you want to add some special consideration to these extreme outliers, consider transforming your loss function: as of now, if going from 0.97 to 0.99 means that the predictions between say 0.2 and 0.8 would shift ever so slightly off the target, the model might "think" of it as an overall deterioration - you might have a different opinion. Make your target function reflect what you actually want out of the model! $\endgroup$
    – Lodinn
    Mar 3, 2022 at 10:06
  • 1
    $\begingroup$ Please someone mention isotonic regression, Platt scaling and beta calibration. $\endgroup$
    – usεr11852
    Mar 8, 2022 at 22:46

3 Answers 3

7
$\begingroup$

This is what I would focus on.

  • Limitations on max_depth might cause terminal nodes to group together observations with very small probabilities with other observations where probabilities aren't that small, so the effect is to move the leaf weights away from extreme values. Likewise, something similar with large probability observations. Try increasing max_depth.
  • lambda penalizes the absolute value of the weights. You want weights with large absolute value, because these weights allow for probabilities closer to 0 and 1, so try setting lambda smaller.
  • Column subsampling could omit the important features (time left in the game sounds important), so I wouldn't use it.
  • Increasing the maximum number of trees dramatically and using early stopping could help.
  • Tuning the learning rate alongside these parameters is important.

Since your question is basically about calibration of probabilities, something to know is that XGBoost is notorious for producing poorly-calibrated predicted probabilities. It's unclear if this is the culprit in your case; usually, the poor calibration arises from predictions that are too close to 0 or 1, but you have the opposite finding here. This is why I think you might be able to close the gap using different hyper-parameters.


I wonder if an XGBoost model is the best approach, because your data are arranged sequentially in time (60, 50, ... 10 minutes remaining, etc.). I would investigate alternative models that can account for this temporal dependency.

If you think about each game as a sequence, the probability of Team A winning should have a wide band around it at the start of the game, and then that band should narrow as the clock runs out. I don't know how to model that, but intuitively, that seems like what you're looking for.

$\endgroup$
6
  • $\begingroup$ Thanks a ton! I'll try out those hyperparameter suggestions. I think my fear was that most of those result in more flexible learning (which makes sense given I'm trying to make the model learn a more particular patter) which could result in overfitting. But worth checking it out! $\endgroup$
    – dfried
    Mar 2, 2022 at 20:02
  • 2
    $\begingroup$ @DanielFriedman You might try to overfit your data first, and then increase the regularization until you find the right compromise. $\endgroup$
    – Sycorax
    Mar 2, 2022 at 20:03
  • $\begingroup$ What alternative models do you think might pick this pattern up well? My thought was that the data is very partitionable (e.g. it makes a lot of sense to look at games with a certain range of score-time combos) so xgboost might work well, but I'm curious to hear any other suggestions! Thanks again $\endgroup$
    – dfried
    Mar 2, 2022 at 20:04
  • $\begingroup$ I'm not a time-series expert, but my guess, without doing any research, is that sports statistics and sports betting folks have done work in this area to look at time-aware approaches to sports outcomes. I'd start with a literature review. $\endgroup$
    – Sycorax
    Mar 2, 2022 at 20:05
  • $\begingroup$ My understanding about the "notorious[ly...] poorly-calibrated predicted probabilities" is that GBMs tend to push probabilities towards 0 and 1 though, which wouldn't explain OP's issue. Underfitting certainly would, so +1 for all the parameter suggestions. That said, maybe also consider a post-processing calibration technique? $\endgroup$ Mar 2, 2022 at 21:59
4
$\begingroup$

If you want precise estimates of probabilities, don't use algorithms based on decision trees. To get a probability estimate of a decision tree, you would count the class occurrences in the node and divide by the node size. This would never lead to a smooth function describing the probabilities, same as regression trees don't produce smooth approximations of the functions. If you consider many trees, this would smooth a little bit more, but never perfectly. This would be especially striking for extreme probabilities as in your case (for a decision tree to predict probability like 0.001 the node would need to contain >1000 samples). As mentioned by Sycroax, it is a poor choice of the algorithm if you care about well-calibrated probabilities.

$\endgroup$
12
  • $\begingroup$ Thanks! Suggestions for alternatives? I'm thinking logistic regression probably isn't flexible enough... could also try neural nets or SVMs $\endgroup$
    – dfried
    Mar 2, 2022 at 20:05
  • 1
    $\begingroup$ Have you tried it? $\endgroup$
    – Tim
    Mar 2, 2022 at 20:08
  • 2
    $\begingroup$ With boosting, the limitation of decision trees "to predict probability like 0.001 the node would need to contain >1000 samples" is not necessarily applicable. $\endgroup$ Mar 2, 2022 at 20:16
  • 1
    $\begingroup$ @DanielFriedman how many wins and loses did you observe for the scenario where you except 0.99 probability? $\endgroup$
    – Tim
    Mar 4, 2022 at 17:34
  • 1
    $\begingroup$ In the last minute of the game I have 2100 observations with home leads of 10 points or more, all of which are wins. The model isn't showing anything above .97 $\endgroup$
    – dfried
    Mar 4, 2022 at 21:05
2
$\begingroup$

As @Sycorax and @BenReiniger pointed out, the problem is that the probabilities are not calibrated (or not as calibrated as well as you'd prefer). Here is how you could calibrate the XGBoost probabilities. Use the following model:

P(y|x) = 1/(1+exp(-(a+x)))

where x is the logit function of the original probabilities produced by XGBoost:

logit = log(p/(1-p))

and y are the same outcomes you are already using. This is based on the paper by van den Goorbergh et al., "The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression", arXiv 2022 (see the Methods section). In my experience works well.

You can implement this in R using this statement:

recal_mod = glm(y ~ 1, offset = logit, family = "binomial")

Note that this model is a logistic regression whereas the independent variable has a fixed weight of 1 and only the intercept is fit. To my knowledge this type of model is not supported in python scikit-learn LogisticRegression() function, so I use R. Note also that it uses lot of RAM for large datasets, so you may want to to downsample your data.

The following graphs illustrate how it works. The first graph is a calibration curve before recalibration. The second graph is the calibration curve after recalibration using the above model. I don't know if it will achieve the calibration that you want but it's worth trying

Calibration curve before recalibration Calibration curve after recalibration using the above model

$\endgroup$
1
  • $\begingroup$ This kind of shift is useful when the prior class distribution is different from the one trained on, e.g. when resampling before modeling. I'm not sure that's the case in this Question? $\endgroup$ Mar 8, 2022 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.