Questions tagged [xgboost]

A popular boosting algorithm and software library (stands for "extreme gradient boosting"). Boosting combines weakly predictive models into a strongly predictive model.

Filter by
Sorted by
Tagged with
0
votes
1answer
27 views
+50

Choosing model from Walk-Forward CV for Time Series

my question is about the ‘correct’ form of analysis following the use of multiple train/test splits on time series data. Specifically, I’m using Sklearn time series split to generate 10 windows for ...
0
votes
0answers
30 views

Output of xgboost() while optimization is not very intuitive

I am running xgboost() on a data set with a data set with below columns. ...
1
vote
2answers
43 views

$R^2$ of Log transformed data is positive, however that of reversed transformed data is negative

I am running an XGBoost model with a continuous target variable. With ~200 features I am getting a Test $R^2$ of 0.54. By looking at the distribution of the target variable, it appears it's highly ...
0
votes
0answers
15 views

Scaled and Unscaled features are giving different feature importances when using LIME

Now, based on my understanding, feature scaling should have no impact on my model results due to the fact that XGBoost isn't sensitive to monotonic transformations. Ref My concern is the model ...
2
votes
3answers
69 views

One hot encoding of a binary feature when using XGBoost

I already asked this question is SO; however, I realized that this may be a better place for this type of question. I am well aware that when using categorical features with tree based models such as ...
3
votes
0answers
14 views

When should one use Bradley-Terry instead of gradient boosted trees for pairwise ranking

Both the Bradley-Terry model and Gradient boosted trees can be used to learn a ranking from pairwise comparisons (e.g. with libraries choix and XGboost). How do they relate to each other? Is there ...
0
votes
0answers
8 views

What can be done to fix this cumulative lift curves

I trained a model and got this lift in testing. I would expect a lift that monotonouslly decreases, but instead i have this What can be inferred of this? I'm using h2o automl, which usually gives me ...
3
votes
1answer
69 views

XGBOOST objective function derivation algebra

I need some help please with the derivation of xgboost objective function. I am following this online tutorial (Math behind GBM and XGBoost) How do you go from here $$ loss = \sum_{i=1}^{n} \left( ...
0
votes
0answers
16 views

Bootstrapped data to fit models and comparison

I have a large data set, say 40k row by 60 columns. Now I do bootstrapping (resample the rows with replacement) on this whole data set for 10k times. Then I used each of the 10k data sets to fit a ...
2
votes
1answer
39 views

Overfitting in extreme gradient boosting

My situation is: 36,197 observations/ 125 outcomes in training data 26 predictors A relatively successful prediction model has been built in a similar dataset using just logistic regression; I ...
-2
votes
1answer
20 views

What algorithm gives more accurate results xgboost or lightgbm?

LightGBM builds trees much faster than XGBoost. Though which algorithm works better in terms of accuracy in general and for specific tasks such as regression, classification in your experience?
2
votes
1answer
12 views

Positive and Unlabelled data classifier

I have few customers and am supposed to find 'similar' people from a pool of customers and non-customers, what approach would you follow to solve such a problem? The key point here is that non-...
0
votes
0answers
40 views

Interpreting hamming loss for multilabel classification

I have a multi label - multi class classifier that aims to predict the top 3 selling products out of 11 possible for a given day. Using scikit learn's OneVSRest with XgBoost as an estimator, the ...
0
votes
0answers
16 views

Average of importance gain for a categorical variable

Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc). I train an XGBoost model on a numeric ...
0
votes
1answer
51 views

training and validation accuracy increasing - XGBoost

I am running 10-folds 10 repeats cross validation over my data. I am using XGBoost Classifier with hyper parameter tuning. The learning curve looks as follows: However, both the training and ...
1
vote
0answers
25 views

Calculate Gini Importance for Boosting Trees

From my understanding, Gini Importance means Mean Decrease in MSE for regression objectives, and Mean Decrease in Impurity for classification objectives. Typical random forest packages like ...
0
votes
0answers
24 views

In XGBoost with a f1_score, is the iteration with a lower or higher score the better iteration?

In the following XGBoost script the output states iteration 0 with score 0.0047 is the best score. I would expect iteration 10 with score 0.01335 to be the better score? Output ...
0
votes
0answers
31 views

machine learning and class imbalance

I'm trying to apply machine learning to classify around 50 diseases according to their protein intensities (ie, disease X is characterized by abnormal levels of proteins a, b, and c). I've tried ...
0
votes
0answers
37 views

How can I implement k-fold CV for my ranger and logistic model in R?

I am currently trying to learn about k-fold CV as up until now I have just used a hold-out sample, and after a little reading decided that k-fold cv would be better for me in many cases. However, ...
0
votes
0answers
13 views

Why more differentiated class distribution got lower feature importance using XGBoost?

I'm trying to use XGboost for fraud prediction, using Average Gain to rank feature importance. The assumption here is, features with higher importance tend to differentiate fraud vs nonfraud better. ...
0
votes
0answers
72 views

How XGBoost use Histogram to Determine Splitting Features?

I'm trying for some time to understanding, the splitting method used in XGBoost to determine the best split. But unfortunately I didn't find any clear explanation of it. I found this post which ...
1
vote
0answers
20 views

XGBoost model in Spark --> Missing value treatment [closed]

Unlike python, where missing value is handled internally by the XGBoost algorithm, While building XGBoost model in SPARK, the missing values are implicitly converted to 0.0(float?!). Is this okay ? ...
1
vote
0answers
22 views

Simplifying XGBoost

I need to explain the concept behind XGBoost to a few people. Although I understand how it works, I'm looking for a good analogy using which I can easily describe XGBoost. Probably something which ...
0
votes
0answers
40 views

Do you use early stopping in xgboost when you have a small sample size?

I'm running xgboost on some simulation, where my sample size is 125. I was measuring the 5-fold cross validation error, i.e., in each round my training sample size is 100 and testing sample size is 25....
1
vote
0answers
18 views

Can we extend the XGBoost algorithm to higher order steps?

In XGBoost, the idea is at every round of boosting we add an additional model (a decision tree in XGBoost for trees). This model is learned to optimize the second order Taylor expansion of the loss of ...
1
vote
0answers
44 views

Violation of the IID assumption in Gradient Boosting

Generally, machine learning methods make little to no statistical assumptions. However, a key assumption they do make is that the data are IID. What are the implications of a violation of the ...
0
votes
0answers
24 views

Aggregating/Summing feature importance for categorical fields in xgboost

When I use a standard GBM package for categorical features, I receive importance metrics at that field level. For example: State: 0.75 However, when using XGB the importance is split by each of ...
1
vote
0answers
68 views

Why do the XGBoost predicted probabilities of my test and validation sets look well calibrated but not for my training set?

I am using an XGBoost classifier to predict propensity to buy. After drawing a calibration curve to check how well the classification probabilities (predict_proba) produced are vs actual experience, I ...
0
votes
1answer
22 views

Interpreting trees of XGBoostRegressor Model

I fitted a dmlc XGBoostRegressor model on a dataset with the variables ['CPI', 'Fuel_Price', 'Temperature', 'Unemployment'] and ...
3
votes
1answer
81 views

Biased prediction (overestimation) for xgboost

I run xgboost and elastic-net on the same dataset for a classification problem, say we have ...
1
vote
1answer
259 views

R - xgbTree, xgbDart and gbm in Caret predict Small Range

I am currently facing an issue in my regression model. I have tried the models in the title (xgbTree, xgbDART, and gbm in caret), and they tend to predict a very small range for the output variable. ...
0
votes
0answers
25 views

Multilevel model method in non-Bayes algo? [duplicate]

I have learned multilevel model in Bayes methods, which I think is really cool. Different levels data could share information which could improve the total model. Is there any similar methods in non-...
0
votes
0answers
39 views

AutoGrad Hessian in custom loss function in XGboost takes very long time

I am using the autograd package and generating the hessian of my loss function using that package as part of my custom loss function XGB model. However, it takes an extremely long time to iterate ...
1
vote
1answer
57 views

What calculation does XGBoost use for feature importances?

Does anyone know what the actual calculation behind the feature importance (importance type='gain') method in the xgboost library is? I looked through the documentation and also consulted some other ...
0
votes
0answers
21 views

Ratio of data matching / not matching target variable in training dataset in XGBoost

I have inherited an XGBoost model and training script. The current implementation uses a training data set of data split 50 / 50 between inputs matching the target variable, and data not matching the ...
0
votes
0answers
95 views

scale_pos_Weight , weights params impact on loss calculations for xgboost and lgbm for unbalanced classes

I went through all the questions here and discussion available on web, to figure out how the 'scale_pos_weight' (and 'class_Weight' for multi class problems & individual instance weights given by ...
0
votes
0answers
34 views

XGboost vs. LightGBM, node splits

I believe by default, xgboost or lightgbm use all the features in the model for splitting the nodes in each tree (is this correct? can opt to select few by colsample options) and can the features be ...
0
votes
1answer
38 views

Transition for scikit learn to xgboost: Where can I find a comprehensive documentation for xgboost? (Python)

As the internet seems to be conviced that xgboost is well worth a shot when working with decision trees anyways, I set out to try it. I deal with a binary classification problem. Up to now, I was ...
0
votes
1answer
34 views

Get individual features importance with XGBoost [closed]

I have trained an XGBoost binary classifier and I would like to extract features importance for each observation I give to the model (I already have global features importance). More specifically, I ...
1
vote
1answer
146 views

Panel Regression vs. XGBoost Time Series Features

Panel regression is a technique to merge longitudinal and cross sectional data together in a linear model. Linear model doesnt work well since by bringing time series features into the model, it can ...
2
votes
2answers
105 views

How to Reduce Number of Variables Before Running Random Forrest or XGBoost

I've simplified the problem I'm working on for this post, so that the focus is on the issue I'm having. I'm trying to predict if a patient will be diagnosed with arthritis in 2019, based on the ICD-...
0
votes
2answers
179 views

How to perform SHAP explainer on a system of models

I have developed a model with Autoencoder + XGBoost. Autoencoder is used to reduce dimensionality and then passed on to XGBoost for prediction. I would like to understand the feature importance of the ...
6
votes
2answers
819 views

gamma parameter in xgboost

I came across one comment in an xgboost tutorial. It says "Remember that gamma brings improvement when you want to use shallow (low max_depth) trees". My understanding is that higher gamma higher ...
1
vote
0answers
29 views

Are boosted machine learning methods robust against low probable feature combinations when predicting?

I would like to use machine learning methods in the potential outcome framework, that is, simulating outcome for all observations under different values of a specific predictor, while keeping all ...
1
vote
3answers
288 views

Binary Classification in Imbalanced Data; Oversampling and Imputation

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle ...
1
vote
1answer
39 views

How can I make XGBoost have an exponentially distributed output?

The input distribution is exponential, but XGBoost's predictions' distribution is always unimodal but not exponential. Is there a way to make it exponential?
1
vote
1answer
22 views

What could cause ML predictions to be multimodal when the inputs are unimodal? [closed]

This appears to be a systematic issue that occurs with the whole range of data points: In general, what are some possible causes for discrepancies like this? I know it's impossible to pinpoint the ...
1
vote
1answer
174 views

Using instance weights in XGBoost

I want to understand whether giving weights to instances across a dataset in XGBoost using the below method makes sense. I switched to this method after trying out a few approaches that didn't fare ...
2
votes
2answers
77 views

Important features for the XGboost algorithm are also the most important for the training of DNN?

I know that both a deep neural network (DDN) and the gradient boosting decision tree algorithm Xgboost can be used for the task of classification. I'm using a DNN first and it works fine. With ...
1
vote
2answers
571 views

Calibration curve of XGBoost for binary classification

I'm working on a binary classification problem, with imbalanced classes (10:1). Since for binary classification, the objective function of XGBoost is ...