5
$\begingroup$

I recently trained an xgboost model for a binary prediction task; the dataset had rougly 900 class 1 and 100 class 0 rows. The model didn't fare too well (AUC 0.64) and none of the features had SHAP value to speak of (below 0.01 for every feature). All the predictions were between 0.49 and 0.51, so I got the impression the model was basically useless, a coinflip if you will.

What got me thinking is that on the same dataset, I also trained two binary classifications using xgboost as well with different target variables, with much better results (AUC ~ 0.75, reasonable feature importances, predicted probabilities all over the range 0-1). All three targets were similiar in nature (patient survey results, all from the same surgical procedure, basically asking the patients in diffeent ways how they felt and how well they function).

Now, I need to find why the model (and also lots of variations I tested) fails on this one particular target variable. How could i do that? The feature variable data seems fine (as it works with different targets); how could I analyze the data quality for the remaining target? Are there other sources where something could have gone wrong? Basically, I'm asking what can be done as a sort of data/model "autopsy" to find what killed my modeling for this particular target variable.

$\endgroup$
5
  • 2
    $\begingroup$ Did you artificially balance the classes? I find it hard to believe that the model is giving all predictions around $1/2$ with the class ratio being $9:1$. $\endgroup$
    – Dave
    Commented Apr 8, 2023 at 12:46
  • 1
    $\begingroup$ I did not balance the classes in any way. $\endgroup$ Commented Apr 8, 2023 at 13:07
  • 1
    $\begingroup$ What is the actual problenlm you try to solve / predict? Without knowing what your regret is and what features you have it's hard to diagnose this ( if your target is a coin flip ... Then this is working as expected) $\endgroup$ Commented Apr 8, 2023 at 14:21
  • 1
    $\begingroup$ It would help if you could edit the question to provide a lot more information about the data. For example, you say "All three targets were similar in nature," but just how highly correlated were they in practice? Why are you modeling the 3 targets separately instead of combining them into a single outcome scale? If you have what seems to be (semi-) continuous target variables, why are you reducing them to binary outcomes? (Or do I misunderstand the targets?) How many and what type of features are you using in your model? Without such details, all that can be suggested is vague generality. $\endgroup$
    – EdM
    Commented Apr 8, 2023 at 14:28
  • 2
    $\begingroup$ Are those predictions around $1/2$ coming from in-sample or out-of-sample data? // @EdM My impression is that the discussion about the other model with three targets is mentioned to show that the OP has been somewhat successful with XGBoost modeling in other situations, even though this one is not successful. $\endgroup$
    – Dave
    Commented Apr 8, 2023 at 14:30

2 Answers 2

1
$\begingroup$

First, note that $AUC = 0.64$ isn't such terrible performance. Sure, it is better to get $AUC = 0.9$ or $AUC = 0.99$, but if you validate $AUC > 0.5$, then your modeling has some ability to tell apart the two categories. Consequently, it is not a given that you really have to do an autopsy to determine the cause of death of your model. Your model sounds like it could be very-much alive.

Since you have a $9$$:$$1$ class imbalance, it cannot be that your predictions between $0.49$ and $0.51$ are calibrated. If all of the events really happened with such probabilities, you would not have that kind of imbalance (your categories would be roughly balanced, since all events happen with about probability $0.5$). Consequently, since the $AUC$ seems to be decent (not great, but decent) and reflective of at least modest ability to distinguish the categories, you might get somewhere by calibrating the predictions.

$\endgroup$
6
  • $\begingroup$ Very likely, the 0.5 is because XGBoost starts from 0.5 and fails to learn anything. $\endgroup$
    – usεr11852
    Commented Apr 11, 2023 at 22:16
  • $\begingroup$ @usεr11852 I would have expected a model that learned basically nothing to at least predict around the prior probability of $0.1$ instead of $0.5$, but maybe I am applying an intuition from logistic regression to a situation where it should not be applied. $\endgroup$
    – Dave
    Commented Apr 11, 2023 at 22:33
  • $\begingroup$ If it makes you feel better, I have a strong suspicion that regression tasks also start at 0.5. :D In LightGBM this has been mostly rectified by using the boost_from_average argument (see discussion here: github.com/microsoft/LightGBM/issues/1336) but XGBoost being the OG that it is, starts from 0.5. Ultimately if there is indeed signal to be learned and we train adequately, after the first (couple/dozens/hundreds) iterations the baseline should be inconsequential. $\endgroup$
    – usεr11852
    Commented Apr 11, 2023 at 23:54
  • 1
    $\begingroup$ @usεr11852 the first tree ought to push the average from 0.5 (in almost every leaf) toward an average of 0.1 though, unless the learning rate is very small? $\endgroup$ Commented Apr 12, 2023 at 2:03
  • 1
    $\begingroup$ @BenReiniger: We do not say something qualitative very different, but often our $\eta$ is small and the dataset noisy (as here) so that "ought to push" won't happen unfortunately. Actually, regarding the "first tree" point that both you and Dave raise: unless our $\eta$ is reasonably large (0.2+), we won't see anything close to probabilities substantial away from 0.45-0.55 if we have a learning rate of 0.1 with our first tree even if we have an imbalanced response. Try training a XGBClassifier(eta=0.1, n_estimators=1) to an imbalanced and check the lowest probability predicted. $\endgroup$
    – usεr11852
    Commented Apr 12, 2023 at 3:30
1
+100
$\begingroup$

Disclaimer: I assume all three models have: 1. Common preprocessing of the training set. 2. XGBoost classifier hyperparameters tuned adequately. 3. Evaluation is done on a separate test set. 4. No target leakage.

Gradient boosting machines (like XGBoost) are great but often they are hard to interpret. Here we can move forward with our investigation in two ways: 1. Use simpler explainable models. 2. Use post hoc explanation methods. Both options involve some visualisation because ultimately we want to have some contrasts. (Third option in the end)

Option 2: I will actually start with the post hoc explanation methods first. Use partial dependence plots (PDPs) from the models you think are "reasonably performant". Examine the ones corresponding to the variables having the highest importance (I would use the usual gain attribute as the average gain across all splits the feature is used corresponds the "closest" to generic feature importance for prediction purposes). These plots should show some coherent gradient/variation. Superimpose those PDPs against the same PDPs from the less performant model. Those plots should immediately tells us how our "performant models" use certain variables while our least performant model cannot find any coherent signal there. Note that it is not necessary that different models have similarly looking PDPs, the contrary is usually true. What we want to show is there one model finds meaningful variation and the other model does not.

Option 1: Using a glass-box model that is directly explainable allows us to immediately ascertain how certain predictions are made or why can't they be made. As a first step: Let's remember that XGBoost ultimately uses CARTs as base learners. That means that we can visualise (some of) these trees and see what is learned. I would suggest first trying a Random Forest (not Extra Trees) with a very small number of trees. Again the catch is to first see how a "good ensemble learner" behaves and contrast that with the "bad ensemble learner". As a second step: Let's also remember that GBMs are closely related to GAMs. We can use the most informative features from the post hoc analysis and build a GLM or a GAM. While this is wrong as a general model-building strategy because it leads to overfitting and fails to capture interactions, the catch here is to see if we can indeed overfit any signal we might have. We can then compare how that signal can or cannot be found between our models (for example, a $\beta_{x_1}$ might be statistically significant in our "good model" but statistically insignificant and/or with a minimal effect size in our "bad model".

A third option I wouldn't immediately consider but one might try is to use Boruta or a similar meta-learning variable importance algorithm. I personally find such methods not horribly insightful but then again they should allow one to quantify feature importance scores with a standardised methodology. We can then make an almost like-for-like comparison too. As above we would want to contrast the variable importance computed rather than focus on the exact magnitudes.

Final comment: The above is essentially a guide to post hoc model explainability comparisons. There are no substitutes for doing proper EDA before modelling. Simply using different response variables and then saying "this sometimes works and sometimes doesn't work" is completely normal. The whole argument here is why seemingly related response variables are not having similar performance. But we have no commentary if these response variables have the same "noise" components too. Again, EDA and literature reviews are our friends here. Good luck! :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.