Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

0
votes
0answers
9 views

How to do variable selection for Gradient boosting models like Xgboost and LightGBM

I am building a classification model with about ~110 variables and that gave me an AUC of about 71.96 on validation. I added about 10 more features and my AUC value decreased to 71.56 (which led to ...
0
votes
0answers
7 views
0
votes
0answers
30 views

Which method is correct for calculating total deviance explained for boosted regression trees?

I have the following output from a boosted regression trees model and I would like to calculate the total deviance explained. ...
0
votes
0answers
20 views

Controlling over-fitting in local cross-validation LightGBM

I am training a lightgbm model on a binary problem (~20% of events) with below parameters: ...
1
vote
1answer
30 views

Binary classification for imbalanced distribution of target/response class for age

I'm trying to build/train model that depends on many attributes where age is the most important one (it has significant impact on AUC). Overall target class count is quite balanced (+40% vs. -60%) ...
-1
votes
0answers
12 views

How to choose which classifier is better than the other one?

I'm doing binary classification by using Gradient Boosting Machine (GBM). First, I used scikit implementation of GBM and get these results: Training Dataset AUC: 0.97 Test Dataset AUC: 0.90 And ...
1
vote
0answers
12 views

How to combine from multiple probability in adaboost? [closed]

I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them? For simplicity, there ...
3
votes
1answer
29 views

Why does boosting accuracy change when run multiple times?

I have a set of 380 observations with 30 predictor variables and one response variable. I have determined using other methods (bagging and random forest) that there is a set of 7 variables that are ...
2
votes
0answers
18 views

What does “pairwise” distribuion in the R gbm package mean? [closed]

In the R gbm function (within the gbm R package), the distribution argument can get the value "pairwise". The help document states: I would like your help in understanding what is that "pairwise" ...
0
votes
0answers
18 views

Why we need to do bootstraping and extract confidence interval for classification problems?

My question is related to bootstraping and extracting confidence interval for classification problems. Let's say I have 25 number of data points and 2 features and use Gradient Boosting Machine (GBM) ...
1
vote
1answer
22 views

Negative Feature Importance Value in CatBoost LossFunctionChange

I am using CatBoost for ranking task. I am using QueryRMSE as my loss function. I notice for some features, the feature importance values are negative and I don't know how to interpret them. It says ...
1
vote
1answer
34 views

Regression when target has a wide range

I'm working on a regression model where I have to predict time. These times go from a few seconds to up to 30 min and more. I calculated the sMAPE through 1 minute bins of the target, and noticed ...
2
votes
0answers
23 views

In gradient boosting, what is the thing being boosted called?

Quoting from this answer that explains how to do boosting when a 'link' function is involved: "Instead, we can re-express this as a function of $L_i$, (in this case also known as the log odds) $$ \...
0
votes
0answers
37 views

Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]

I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
0
votes
2answers
34 views

why is number of epochs set as external parameter?

I am confused by the very notion of epochs in neural networks (as well as number of trees in gradient boosting). Gradient descent method (as most optimization algorithms) keep going until the loss ...
0
votes
0answers
25 views

How to convince people that developed predictive model based on Gradient Boosting Machine (GBM) has enough accuracy?

First of all, I'm not a data scientist. I'm an engineer that wants to use machine learning to do a binary classification based on a data that is extracted from computational modeling. I have four ...
2
votes
1answer
44 views

The error keeps decreasing with the increase of number of trees

I want to find optimized number of trees in Gradient Boosting. However, the error keeps decreasing with the increase of number of trees, I set the number of trees over 1000, but the error still keeps ...
1
vote
0answers
40 views

How does offset in XGBoost is handled in binary:logistic objective function

I am working on a mortality prediction (binary outcome) problem with “base mortality probability” as my offset in the XGboost problem. I have used gbtree booster and binary:logistic objective function....
0
votes
0answers
41 views

what is the formula for r2 when using H20 GBM H2OGradientBoostingEstimator

I used a H2OGradientBoostingEstimator to do a classification (n features into a binary 0/1). What does the reported R^2 mean? Should I look more into AUC or into R^2? And for classification, what ...
0
votes
0answers
97 views

Variable importance in a GBM

I have build a model with a Gradient Boosting Machine (GBM) and calculated the feature importance. All features are factors. Now, I know which features are most important. However, the features have ...
2
votes
1answer
37 views

Caret: gradient vs gam boosting

What is the difference between a boosted additive model (e.g. caret model: gamboost) and a general stochastic gradient boosting model (caret model: gbm)? A gradient boosting model is additive by ...
1
vote
1answer
42 views

Updating decision tree for new data

Lets say you have trained a decision tree for 40 gigs of data. On Monday morning you receive 10Gig new data and produce some results quickly to report to your boss. Can you update the decision tree ...
4
votes
2answers
173 views

Is it ever recommended to use mean/multiple imputation when using tree-based predictive models?

Everytime that I am making some predictive model and I have missing data I impute categorical variables with something like "UNKNOWN" and numerical variables with some absurd number that will never be ...
0
votes
1answer
38 views

Defining the cases where Neural Networks outperform tree-based methods

It is well-known that neural networks are currently superior to most of the alternatives to do prediction from images (with CNNs) and sequential data (RNN, transformers...). However for other ...
2
votes
1answer
35 views

A regressor failed to learn extreme values

I am working on a regression problem using xgbclassifier (https://xgboost.readthedocs.io/en/latest/python/python_api.html) The output values range from 0 to 10 (log-normal distribution), but when I ...
2
votes
1answer
31 views

Why are AUC and logloss metrics not available in the “maximum metrics” table produced by H2O? [closed]

I am running the h2o.gbm algorithm using five-fold cross validation to predict a binary outcome. I want to see what threshold to use as a cutoff for classifying predictions, and I am wondering why the ...
0
votes
1answer
26 views

Gradient boosting understanding of residual picture

I recently looking at the Gradient boosting using following blog https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d I try to understand the picture but I need some help For ...
0
votes
1answer
65 views

How are the training and cross-validation metrics calculated in H2O?

I am working with the GBM algorithm in H2O in R. I am using 100% of the data as the training data, and then using 5-fold cross-validation to train and validate my model using 100% of the data. My ...
5
votes
2answers
194 views

Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
0
votes
1answer
69 views

Can AdaBoost be used for regression?

I know that AdaBoost can be used for classification, but how about regression? With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) ...
1
vote
1answer
114 views

What is an intuitive interpretation of the leaf values in XGBoost base learners?

I'm learning XGBoost. The following is the code I used and below that is the tree #0 and #1 in the XGBoost model I built. I'm having a hard time understanding the meanings of the leaf values. Some ...
2
votes
1answer
572 views

Classification XGBoost vs Logistic Regression

I have a binary classification problem where the classes are slightly unbalanced 25%-75% distribution. I have a total of around 35 features after some feature engineering and the features I have are ...
0
votes
0answers
40 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
0
votes
0answers
128 views

Feature selection in xgboost vs GBM in H2O

I am working on a big data set( more than 100 variables) and 30 million observations. I tried to build 100 models with a grid search using both XGBoost and GBM in H2O (Sparkling Water). I realized ...
2
votes
1answer
125 views

How to interpret chart generated by gbm.perf function?

I'm new to GBM.Can you help me to understand the interpretation of gbm.perf function? I used following code in R best.iter = gbm.perf(train, method="cv") & got ...
0
votes
1answer
64 views

Are the same number of trees required to compare Random Forest against GBM?

My training set has 13,737 observations with 53 predictors. I need to compare the accuracy of Random Forest vs. GBM. For Random Forest, I set ntree = 128 [based on ...
0
votes
0answers
19 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
1
vote
0answers
41 views

XGBoost tree dump contains lots of empty trees

After fitting a regression model using XGBoost, I want to inspect the individual trees that were built. In the resulting table, I find a lot of 0-depth trees, i.e. trees with only a leaf node, and ...
1
vote
0answers
32 views

Evaluation of Classifier Performance on Imbalanced Dataset with Lift Chart

I trained a classifier on imbalanced dataset (label={0,1}) by assigning higher weight to rare event(label=1). Lift chart shows that the predicted and actual curves are very separated. I also trained ...
2
votes
1answer
22 views

binomial responses in h2o gbm

I am modeling the probability of success in a dataset where I have a both the number of trials and the number of successes (and, obviously, I am modeling $p_i=\frac{total successes}{total trials}$). I ...
2
votes
1answer
27 views

Learning Rate impact on model building time

I wanted to know that does learning rate impact the model building time in case of Gradient Boosted Trees. I do understand that increasing the number of trees have an impact( more the trees, more the ...
2
votes
2answers
55 views

Why are all predictions made by XGBoost distinct?

If I understood correctly the XGBoost is a framework that operates on gradient tree boosting. It means that behind the scenes, it uses a decision tree to make a prediction. So, from what I read in the ...
1
vote
1answer
14 views

Understanding `score` in LightGMB

I'm newly introduced to the LightGBM for a regression problem. Having read the documentation of LightGBM (here), I got puzzled about the ...
1
vote
0answers
58 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
0
votes
0answers
27 views

How do Gradient Boosted Trees calculate errors in classification? [duplicate]

I understand how gradient boosting works for regression when we build the next model on the residual error of the previous model - if we use for example linear regression then it will be the residual ...
2
votes
0answers
39 views

Implausible variable importance for GBM survival: constant difference in importance [closed]

I have a question about a GBM survival analysis. I'm trying to quantify variable importances for my variables (n=453), in a data set of 3614 individuals. The resulting graph wi th variable importances ...
0
votes
0answers
25 views

ada model- variables overall importance

I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees. I would like to have a ...
1
vote
1answer
23 views

Whats is the difference between using risk() and cvrisk() in the R package mboost

I am currently running an additive model using the function gamboost() in the package mboost. When using the ...
0
votes
1answer
55 views

Why AdaBoost works exactly the way it does

I understand the basic idea of AdaBoost -- when training weak classifiers, use more of the difficult examples. However, it puzzles me why I sould modify the weights the way AdaBoost does. There are, ...
0
votes
0answers
31 views

Conditions for Adaboost to perform well

Under which conditions does the AdaBoost algorithm yield good results even on weak learners (i.e. slightly better than random classifiers)?