Skip to main content
8 votes

Can XGBoost learn more complicated interactions/features?

You are correct. This strongly depends on the amount of data and the nature of the data too. The functional form of $y = a/c + \epsilon$ can actually be quite tricky and in the case of a booster, we ...
usεr11852's user avatar
  • 44.7k
7 votes

Do Boosted tree models result in only one final tree?

As the other answer mentions, the direct result is a collection of trees. Depending on the operation used to combine trees, however, these can be combined into a single tree spanning the whole ...
Firebug's user avatar
  • 19.4k
5 votes

Why is there no improvement when training Xgboost with pseudo-Huber loss?

There is no definite answer at this but I would note one major and one minor point: The major point is that: A XGBoost booster starts with a base_score. That is ...
usεr11852's user avatar
  • 44.7k
5 votes

SHAP algorithm for feature selecion

SHAP probably is not as useful as you would like. In a keynote address to "Why R?", Frank Harrell discusses how feature selection is a mirage. While his simulations do not address SHAP in ...
Dave's user avatar
  • 64.2k
4 votes
Accepted

How to interpret the deviance plot by boosting models

Deviance is a measure of model quality typically (but, I suppose, not necessarily) related to the likelihood. The lower the deviance, the better the model fit. Perhaps think of it this way: models are ...
Dave's user avatar
  • 64.2k
4 votes
Accepted

Help with Classification model for S&P500

There’s a huge issue in what you’re doing that need to have attention drawn to it. I also created a weight variable which is the absolute value of the return, which I would use as input for xgboost. ...
Dave's user avatar
  • 64.2k
3 votes

Are XGBoost probabilities well-calibrated?

XGBoost is well calibrated providing you optimise for log_loss (as objective and in hyperparameter search). ML models tend to "default" to overfitting (as opposed to eg logistic regression, ...
seanv507's user avatar
  • 7,010
3 votes
Accepted

XGBoost Probabilities are unrealistic

You already addressed the problem you were facing by changing the loss function, but as you can learn from Are XGBoost probabilities well-calibrated?, the probabilities returned by XGBoost are in ...
Tim's user avatar
  • 139k
3 votes

XGBoost - Linear Tree

The gblinear booster is an ensemble of generalised linear regression models that is trained using (variants of) gradient descent. As such the concept of a leaf or ...
usεr11852's user avatar
  • 44.7k
3 votes
Accepted

Multivariate Time Series dataset preparation

The correct way to use your input data depends on the way it was collected. In general, you can only use features for which the values are known before the model should forecast and this is usually ...
picky_porpoise's user avatar
3 votes
Accepted

Training a model where true value is only known in batches

TLDR: You could this with a standard ML method (like xgboost) assuming fixed batch size but it is likely to be extremely data inefficient. One could code up a Tensorflow model relatively easily that ...
Cliff AB's user avatar
  • 21.3k
3 votes
Accepted

BerHu custom loss function for XGBoost

Looking at the pseudo-Huber loss, given by (to use R code): pseudo_huber <- function(x, delta) delta^2 * (sqrt(1+(x/delta)^2) -1) you can see how that gets the ...
Björn's user avatar
  • 33.2k
2 votes
Accepted

How to use priors to impute values at an individual level and replicate a distribution of the population?

You could model the column as a latent variable and use MCMC to model the latent variable as a Gaussian informed by your true mean & standard deviation. The data used to train your xgboost model ...
prijatelj's user avatar
  • 446
2 votes

XGBoost Calibration for weighted loss function

If you are doing the one vs. rest approach you are essentially doing calibration for a binary problem. The idea behind this is to get predictions which are as close as possible to the conditional ...
picky_porpoise's user avatar
2 votes
Accepted

Double Machine Learning: What kind of "naive" estimator do the authors use to get such a bias?

I think I figured the answer to my own question. Apparently, what they did was estimating $E[Y|D, X]$ (i.e. fitting a model using $(D, X)$ as features), and then computing the estimator of $\theta_0$ ...
D F's user avatar
  • 741
2 votes

Using XGBoost to learn insights on data itself?

If you want "insights" you should use a model that is interpretable by itself. When you fit XGBoost, it learns to approximate the distribution of the data. When you use SHAP to explain the ...
Tim's user avatar
  • 139k
2 votes

Ordinal vs multinominal classification in XGboost: differences in one-hot encoding

I think you are hitting undetermined behaviour here as XGBoost is not designed to have y be a pandas.DataFrame. I suspect that ...
usεr11852's user avatar
  • 44.7k
2 votes
Accepted

Ordinal log-loss in a multiclass classification in XGBoost?

For a start, you these packages (whether we're talking xgboost or lightgbm or whatever) are almost certainly working with non-softmaxed logits. You can see that by e.g. getting them for a trained ...
Björn's user avatar
  • 33.2k
2 votes

approches for linear extrapolation of xgboost model tails

In general, tree-based models are not exceptional interpolators because effectively they make a recursive space partitioning with step functions of constant value. An option would be to use LightGBM's ...
usεr11852's user avatar
  • 44.7k
2 votes

Overfitting GBM by simultaneously adding trees and lowering learning rate?

Just to note, a very large learning rate can result in underfitting too. Returning to your main question: "yes". Lowering the learning rate while increasing the number of trees will likely ...
usεr11852's user avatar
  • 44.7k
2 votes
Accepted

How does Cross Validation work in decision trees (or tree ensembles)

Requested from comments: Cross-validation is the step for selecting the model including setting its hyperparameters: it is not about training the final model. So it could include the decision tree ...
Henry's user avatar
  • 40.2k
2 votes
Accepted

XGBoost: does manipulating the sample make it "extrapolate"?

If there is no information that allows your system to learn when spikes will occur, it will not predict spikes, no matter what you do. This is just another way of saying that you always have ...
Stephan Kolassa's user avatar
1 vote

Tuning the learning rate parameter in GBDT models

Your intuition is on point, but as you correctly recognise, there is a critical assumption of "the number of iterations is increased accordingly". This assumption might be not satisfied for ...
usεr11852's user avatar
  • 44.7k
1 vote
Accepted

XGBoost original paper equation simplification

Regarding the original paper, I think you are correct. Regarding Wikpedia: it seems that the term was originally $\left[-\frac{\hat{g}_m(x_i)}{\hat{h}_m(x_i)}-\phi(x_i) \right]^2$ but someone edited ...
angryavian's user avatar
  • 2,328
1 vote

XGBoost Learning to Rank with XGBClassifier

I've asked about this on the XGBoost forum, but also wondered if anyone here had any insight into whether using XGBClassifier with objective='rank:map' is actually equivalent to using XGBRanker with a ...
Michael Grogan's user avatar
1 vote

Is there a way to enforce factor importance in random forest/xgboost

If you want to manipulate model into displaying desired feature importances regardless of the importance, that would have resulted from an honest training, you may try to tinker with Cost Efficient ...
forveg's user avatar
  • 51
1 vote

Why is the initial prediction of gradient boosting classifier based on log of odds? Why can't we work with probabilities instead?

For the usual reasons that generalized linear models tend to work with logit-probabilities (=log-odds): You add up terms (in the generalized linear model case coefficients $\times$ predictor values, ...
Björn's user avatar
  • 33.2k
1 vote

Working with subsets of values from single category in XGBoost

Unless you want "cat/lizard/snake" as an extra category (probably not attractive because of many possible combinations, but perhaps an option in some cases), the standard category handling ...
Björn's user avatar
  • 33.2k
1 vote
Accepted

Poor classification even after oversampling minority class

There are few things that could be possibly negatively affecting your model's performance: Your confusion matrix shows a 120:49 negative:positive labels among your test data, which is relatively even ...
Kevin's user avatar
  • 54
1 vote

GBM: Predict the response variable measured in {0,20}

This cannot really be answered without knowing what your integer variable represents. But, assuming it has at least an ordinal meaning, you can try ordinal regression. Many posts on this site, you ...
kjetil b halvorsen's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible