Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [xgboost]

A popular boosting algorithm and software library (stands for "extreme gradient boosting"). Boosting combines weakly predictive models into a strongly predictive model.

0
votes
0answers
7 views

Getting confidence level for xgboost prediction?

I am looking for a solution that can bring confidence level for xgboost (https://xgboost.readthedocs.io/en/latest/) I checked some tools such as https://github.com/donlnz/nonconformist or https://...
3
votes
1answer
57 views

what does regularization mean in xgboost (tree)

In xgboost (xgbtree), gamma is the tunning parameter to control the regularization. I understand what regularization means in <...
0
votes
0answers
6 views

How to use seasonal features in time series regression with models such as xgboost?

I have a hard time understanding how one can create seasonal indices such as a yearly mean or (x - yearly mean(x)) and use them as predictors for monthly n horizon forecast. For example: I want to ...
0
votes
1answer
30 views

How to deal with overestimation of small values and underestimation of high values in XGBoost? [on hold]

I'm running XGBoost to predict prices on a cars dataset, I was wondering what alternatives are there for this kind of problem ...
0
votes
0answers
11 views

Why is XGBoost prediction proba so concentrated within specific range? (unbalanced class)

I am pretty to new to Machine Learning. I am training on some past Kaggle competitions including the Santander Customer Satisfaction Challenge (https://www.kaggle.com/c/santander-customer-satisfaction)...
2
votes
2answers
43 views

Why are all predictions made by XGBoost distinct?

If I understood correctly the XGBoost is a framework that operates on gradient tree boosting. It means that behind the scenes, it uses a decision tree to make a prediction. So, from what I read in the ...
1
vote
0answers
29 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
0
votes
0answers
12 views

Modeling multiple outputs - one model or several

Recently at work I enter an interesting discussion that I thought could continue here and receive your output. I'm trying to model some data that have as an output a categorical variable (let's say X)...
0
votes
1answer
15 views

How to train XGBoost Classifier with soft output distribution

Please correct me if I am wrong. Is it possible to train XGBoost Classifier on soft output? Usually, the output of the model is (N, 1) in dimension which ...
0
votes
0answers
25 views

How would one debug a machine learning model that has a bias?

I'm predicting values roughly forming a normal distribution with mean 0. However, my machine learning model tends to predict lower than 0 on average. I didn't run any statistical tests, but it's very ...
0
votes
1answer
100 views

Running XGBoost with *highly* imbalanced data returns near 0% true positive rate. Tried SMOTE and it did not improve much. What else can I do?

I'm using XGBoost on a dataset of ~2.8M records of hard drive failures, where less than 200 are tagged as failures. After cleaning, there are 11 features in this dataset. Below is my ...
1
vote
1answer
39 views

(Low cardinality) categorical features handling in gradient boosting libraries

In some popular gradient boosting libraries (lgb, catboost), they all seems like can handle categorical inputs by just specifying the column names of the categorical features, and pass it into a ...
2
votes
1answer
42 views

Xgboost and repeated measures

I am learning xgboost and am planning on running a tree model. My dataset includes repeated measures. In a GLMM I would include the ID to account for repeated measures and I'm curious if I should do ...
0
votes
0answers
29 views

How do I include new features that did not exist before into an existing model?

I have a binary classification model predicting sports result with features covering 10 years worth of matches. However, how would I feed new tracking data that is only limited to the last 3 years. ...
1
vote
0answers
13 views

Regression - many samples have the same target

I have a machine learning problem in which I have a many-to-one relationship from samples to targets. I have ~3k samples but only 11 targets with a shared key YEAR ...
1
vote
1answer
21 views

For a specific dataset do all the features have the same importance across different algorithms?

I wonder if by implementing a feature selection technic using training with a specific algorithm you can select the feature you need to use with other algorithms also. To be more specific after I ...
1
vote
0answers
51 views

terminal values in XGBoost / gradient boosting models

I am writing a follow up question regarding a closed Cross Validated question previosuly. The original question can be found here To give a breif overview, I am using the following code to produce a ...
0
votes
0answers
17 views

Variance of error term is nonconstant between observations

I used XGB algorithm to train a model. The task is to train models to predict human personality based on his/her personal photo. We found some significant features when we extracted them by Pearson ...
0
votes
0answers
19 views

GBDT- randomized repetition feature selection

Consider the following approach for feature selection in the specific case of gradient boosting decision trees: Randomly pick X% of features Run algorithm Record importance of each feature Repeat ...
1
vote
1answer
121 views

XGBoost tree “Value” output: [duplicate]

Using the following R code I obtain a decision tree using the agaricus dataset: ...
0
votes
0answers
27 views

How to prevent GBDT from splitting on uninformative features?

I'm looking into using feature importance scores from GBDT for feature selection. Although GBDT does not need manual feature selection, the number of features is a restriction of the production system ...
1
vote
0answers
75 views

Validation data considerably worse performance than stratified kfold cross validation

I've been working on a binary classifer for a NBA spread dataset and am running into an issue where the validation data has considerablly lower performance than the test/train cross validation. I am ...
2
votes
0answers
199 views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
1
vote
1answer
148 views

Weak learners for XGBoost with Tweedie distribution

Could you please explain what are the standard weak learners for XGBoost when the objective parameter equals reg:tweedie? Are they GLMs (with Tweedie distribution of dependent variable) on all ...
1
vote
1answer
38 views

Why do my XGboosted trees all look the same?

I am running an XGBRegressor that is supposed to predict a certain reward associated to different actions, which are one-hot encoded. For testing purposes I am using a small depth=2 and only 10 trees: ...
0
votes
1answer
119 views

xgboost, min_child_weight with eval_metric : 'merror'

I am trying to implement xgboost for a multiclass classification problem. I am using merror as the eval metric. I get a merror of 0.6 and mlogloss of 1.06. If I go on increasing my min_child_weight (...
2
votes
3answers
86 views

how to obtain the 'formula' of a ML classification model

I have followed some tutorial for classification using xgboost, the tutorial usually finishes at predict using the model, and evaluate its accuracy. My question is, how do I extract the 'equation' ...
2
votes
1answer
40 views

Theoretically can gradient boosting achieve 100% of accuracy in an arbitrary dataset?

Consider gradient boosting like gbm or xgboost. I have a labelled dataset (X,y). If I don't care about over-fitting and I allow gbm or xgboost grow as much as needed, eventually can I reach the ...
10
votes
1answer
383 views

Boosting AND Bagging Trees (XGBoost, LightGBM)

There are many blog posts, YouTube videos, etc. about the ideas of bagging or boosting trees. My general understanding is that the pseudo code for each is: Bagging: Take N random samples of x% of ...
1
vote
2answers
79 views

Using the standard deviation in Cross Validation

I'm running a Grid Search to find the optimal parameters for xgboost via sklearn. I can see that the grid search picks the set of parameters with lowest mean MSE. The problem is that upon inspecting ...
1
vote
0answers
40 views

xgboost with 3500 features

I'm trying to make a solution for: https://www.kaggle.com/c/two-sigma-financial-news/ This question relates to using only market price data to predict future prices (i.e. not news data, as specified ...
0
votes
1answer
77 views

Issues with XGBoost on H2O environment

I have a dataset from which I built lags at different levels to use as features in the XGBoost model. When I ran XGBoost models on H2O, the model is picking up the features which contain higher values ...
1
vote
0answers
78 views

How to do classification in mixed effect models in python. My data is nested into groups with binary outcome

Lets say I have 10 sellers (S1-S10). Each seller has 7 buyers which are different for each seller (B1-B7 for S1, B11-B17 for S2 and so on). Each Seller buyer combination has a product category (P1, P2....
0
votes
0answers
127 views

Improving prediction accuracy with XGBoost

I have a 32x20 matrix for which I am trying to use XGBoost (Regression). I am looping through rows to produce an out of sample forecast. I'm surprised that XGBoost only returns an out of sample ...
0
votes
1answer
290 views

Cross Validation Results Interpretation (XGBoost model)

I have a regression model using XGBoost that I was getting great MAE and MAPE results on my test dataset. mape: 2.515660669106389 mae: 90591.77886478149 ...
1
vote
0answers
52 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
2
votes
1answer
379 views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
1
vote
0answers
60 views

xgboost performing poorly

I'm rather new when it comes to using XGBoost but I thought it would be good to learn how to use it. The dataset that I tried it on has 635 data points and 150 features. However the performance is ...
0
votes
0answers
35 views

Alternatives to Neuronal Nets and Gradient Boosting for undifferentiable objectives

It may occur that one has to solve a ML problem but wants to achieve the best result w.r.t a metric that may not be differentiated. This directly implies that such a metric may not be passed to ...
0
votes
0answers
30 views

Difference in runtime and accuracy of the same XGB training code in identical laptops

We are encountering this strange problem and have been unable to figure out the reason. Any help, suggestion will be appreciated. We have two identical laptops (office issued brand new HP laptops ...
2
votes
2answers
105 views

Variables reduction required for Random Forest, Boosting, L1, L2 regularization

I have close to 10,000 variables. I know how random forest/XGB picks number of variables randomly for building the tree. Also regularization takes care of significance of variable by its coefficient. ...
1
vote
1answer
54 views

Random Forest - Is it a good approach to bin categories to reduce the size of the model?

I have a dataset with several categorical columns which I was planning to flatten into binary categories. Let's say I have three features in my dataset. Feature1: has got 300 different values (...
1
vote
0answers
31 views

Decision Tree - What is more efficient a few nodes with hundreds of branches or hundreds of binary nodes (2 branches)?

To make it simple, let's imagine a single category which can have 500 different values. For example city: nameCity1, nameCity2, ..., nameCity500. What is more efficient, to have one single decision ...
1
vote
0answers
167 views

Dealing with categorical feature for xgboost using sagemaker

Currently, I have a dataset which contains 200,000+ datapoints and it contains 20 features with ~10 features as categorical. These categorical columns are countries, state, localities which contains >...
2
votes
1answer
194 views

DQN with XGBoost

Normally a DQN, uses a neuronal network to estimate the Q-Value. I have framed my problem as a regression problem before and have observed that XGBoost does outperform a NN. Is it possible to replace ...
2
votes
2answers
330 views

Why is regression with Gradient Tree Boosting sometimes impacted by normalization (or scaling)?

I read that normalization is not required when using gradient tree boosting (see e.g. https://stackoverflow.com/q/43359169/1551810 and https://github.com/dmlc/xgboost/issues/357). And I think I ...
2
votes
1answer
779 views

XGBoost implementation for unbalanced data using scale_pos_weight parameter

I have a confusion regarding how cost sensitive custom metric can be used for training of unbalanced dataset (two class 0 and 1) in XGBoost. Metric: Cost = 10*#of false positives + 500*# of false ...
0
votes
0answers
75 views

Lag of dependent variable as a explanatory(independent variable) for sales forecasting

I am working on a kaggle competition https://www.kaggle.com/c/competitive-data-science-predict-future-sales/kernels ...
0
votes
1answer
96 views

Balanced data, but unbalanced result

I'm fairly new to data science. I have a multi-class classification problem with 4 classes, 100K rows. The problem is that the classes are balanced but the prediction results are not. (All 4 classes ...
0
votes
1answer
135 views

Is Gradient Boosting Regression Tree able to learn linear models

Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in [0,1]$. Assume one use gbdt to learn this linear model. And if ...