Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Data Considerations for GBM

I am using data summarized in SQL and having a continuous target variable with the weights= option being used since, well, the data is summarized. And each variable combination (just over 15,000) has ...
1
vote
1answer
9 views

How is first tree in Boosting constructed

How is the first tree in GBM constructed, and how the node splitting criteria for the first tree is decided. Can someone please explain, what we are predicting for the first tree (even assuming a ...
2
votes
1answer
21 views

In XGboost are weights estimated for each sample and then averaged

The weights in XGBoost are determined by gradient boosting. So, each sample gets a weight and as each leaf has multiple samples, initially each leaf has multiple weights. But, as a single weight is ...
0
votes
0answers
12 views

Can I build deviance residuals from an XGBoost model that learns an exponential family parameter?

I'm taking a course on GLMs after a few years of using machine learning models. The good about GLMs is how the probabilistic model ties in with the estimation and evaluation. So I'm trying to transfer ...
0
votes
0answers
20 views

I Have A Doubt Regarding AdaBoost Weight Update Rule As Various Sources Cite Two Different Things

I was reading about AdaBoost from Hands-On Machine Learning with Scikit-Learn and TensorFlow and came across the formula used for updating the weights. In the book, it was the following: But in this ...
0
votes
0answers
9 views

What does the loss function adjust in gradient boosting

I get that a gradient boosting algorithm adds weak learners together in order to fit some data. I also get that each subsequent learner after the first one fits the residuals of the current model. ...
0
votes
0answers
28 views

KL Divergence Between Ground Truth and Prediction

I've got four (non-linear, tree-based) models in production and using the average of them as the served prediction. We get ground truth data immediately. During training the optimized candidate models ...
0
votes
2answers
26 views

Feature importance changes drastically when adding other features

I have a model (GBDT) where adding a feature X is not important (according to SHAP), but when I add other features, and add X again, now feature X is the second most important! What could explain that?...
1
vote
1answer
50 views

Partial dependence plot, GBM multinomial

I'm using gbm package to create a gradient boosting model with multinomial family. I have problems in plotting and interpreting partial dependence plots with this type of models. an example: ...
0
votes
1answer
28 views

What is the impact of a dummy variables to boosted trees?

I am currently reading the book "Random Forests" by Yu. L. Pavlov. Then it came across my mind the question If I were to use ensembled tree, say XGBOOST, do I need to transform each ...
2
votes
1answer
31 views

How Gradient Descent is used for classification with Decision Trees?

I'm not able to see how do we use gradient descent to minimize the loss of binary classification with decision tree. What I understood is that we first have a model (decision tree) that try to predict ...
0
votes
0answers
19 views

Is my understanding and presentation of concept of Gradient Boosting correct?

Initially the model is trained with a training set $\{x_{i}, y_{i}\}_{i=1}^{n}$ by minimizing a differentiable loss function $L(y, F(x))$, and, is initialized with a constant value, \begin{align*} F_{...
0
votes
0answers
13 views

Different outcomes between XGBoost and GBM observation weights

I've noticed that the weights argument in the xgboost and gbm r packages don't have the same effect. I had hoped to move from gbm to xgboost for performance reasons, but am not sure how to do so given ...
0
votes
0answers
16 views

Zero inflated Count Data treatment with XGBOOST

I am planning to run an xgboost in response data that is: Count data (0 to 15) Very right skewed Zero inflated (lots more zero than other counts) In the XBG package with R, I have specified count:...
1
vote
1answer
46 views

Is a high learning rate irrelevant when dropping the first or last tree in a GBDT with 100 trees?

Suppose we've trained a GBDT model with 100 trees with a fairly high learning rate. Consider two cases: We drop the first tree in the model We drop the last tree in the model We then compare models ...
0
votes
0answers
15 views

What is this symbol in Adaboost algorithm (SAMME)?

According to the original paper of SAMME algorithm (Adaboost for multiclass problems), it is described as follows: The general idea is very straightfoward, except for this symbol: The author didn't ...
1
vote
1answer
46 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
0
votes
0answers
19 views

Question on Model Validation and Interpretation

I’m a beginner here who is trying to build a model for my personal project (albeit using data I got from my firm). The data is a call center dataset , and Im trying to predict the number of successful ...
1
vote
1answer
46 views

linear weak learners for Xgboost

I see now that Xgboost documentation only considers trees as weak learners, but I remember well tath linear models were an option too, I wander if they are still supported. Anyway, I always assumed ...
0
votes
1answer
25 views

In Gradient Boosting over Decision Trees what does it mean to reconstruct the residual

So I am trying to understand what happens in GBDT and particularly I want to know what it mean to "reconstruct the residual." The way I understand it is that the next tree will use the ...
0
votes
1answer
14 views

Initializing gradient boosting with the sample mean

For gradient boosting in the regression setting, the final vector of fitted values is $$F_M(x) = \bar{y} + \rho_1h_1(x) + \ldots, + \rho_M h_M(x)$$ Suppose I have a new data set $x_{new}$ that I'd ...
1
vote
1answer
178 views

Loss function in for gamma objective function in regression in XGBoost?

Suppose I want to predict $y$ from a set of predictors $x$. $y$ is gamma distributed, so I want to use gamma regression with XGBoost. The help page of XGBoost specifies, for the objective parameter (...
3
votes
1answer
44 views

Tricks for getting NNs to match the performance of GBDTs

I'm working with a tabular dataset with mostly dense features (around 40) and a few low cardinality (meaning around 10 possible values) categorical variables (around 20). In my experience, neural ...
1
vote
1answer
62 views

Why does LightGBM Classifier gives some folks a probability of 1 of belonging in a class with log-loss?

I'm trying to use the LightGBM package in python for a multi-class classification problem and I'm baffled by its results. For a minority of the population, LightGBM predicts a probability of 1 (...
0
votes
1answer
78 views

The Hessian in XGBoost loss function doesn't look like a square matrix

I am following the tutorial for a custom loss function here. I can follow along with the math for the gradient and hessian, where you just take derivatives with respect to ...
1
vote
0answers
17 views

Can this closed form solution of AIC for OLS be applied to tree regressors?

Gordon (2015, p. 201) has a simple clossed form solution for AIC in the framework of OLS $$\text{AIC} = n\log\left(\frac{SSE}{n}\right) + 2k$$ where SSE is the standard $SSE = \sum_i(y_i - \hat{y_i})^...
0
votes
0answers
11 views

How does Adaboost increase the weight of the Data Instance in case of Regression

I know how the weights of the data Instances increases for all the data which were wrongly predicted in case of AdaboostClassifier, however i did not understand how the Weight of the data point ...
2
votes
1answer
19 views

Does it make sense to obtain the greatest error when evaluating only dataset with the most important categorical feature?

I'm running a Gradient Boosting Regressor using scikit-learn. Within my features, I have a categorical feature (let's say Res), ...
0
votes
1answer
28 views

GBDT predict() sometimes gives different class value than using apply() and then sum leaf values [closed]

In sklearn GradientBoostingClassifier, when I use predict() to classify: gbdt = GradientBoostingClassifier(n_estimators=7) tree_preds = gbdt.predict(X) gives ...
0
votes
0answers
38 views

Random Forest doesn't converge in caret

I'm using some synthetic data to try and understand the performance of different ML algorithms on my real data. I'm finding that RF is consistently overfitting the training data even though its ...
0
votes
0answers
26 views

Validation metric for gamma regression

I have a regression task to predict the loan default amount in the case of a default (which is always positive). I am using Gamma regression for this in LightGBM. ...
1
vote
2answers
65 views

Adaboost — how does reweighting affect the learning process for the subsequent learner?

In Adaboost, when you reweight the samples, how does the training process for the next classifier in the boosting algorithm take in to account the weights? Is it reflected in the loss function of the ...
0
votes
0answers
17 views

When is Training data correlation score too low when using boosted regression tree

I'm attempting to use boosted regression trees to look at the effect different fish groups have on coral using the dismo package in R following Elith 2008 and 2017 However the training data ...
2
votes
1answer
35 views

Reference request: consistency of gradient boosting methods

I'm looking for work done on showing the consistency of gradient boosting methods such as gradient boosted decision trees. Friedman's original work only introduces the algorithm, but does not provide ...
0
votes
0answers
28 views

Simple way to know how many models your dataset needs

I have a dataset and was wondering if there is a simple way to know how to break it up so that different models can be used for different subsets based upon the underlying mechanisms. Say I have a ...
2
votes
1answer
71 views

XGBoost CV GridSearch vs Early Stopping

I am using the XGBoost Gradient Boosting Algorithm for a sales prediction dataset. I am planning to tune the parameters regularly with CVGridSearch. Now I am wondering if it makes sense to still ...
0
votes
2answers
42 views

Is boosting and bagging only relevant in the context of decision trees?

In the documents I've seen on boosting and bagging, it seems that they're always talked about in the context of decision trees. What are some other methods in which the two are applicable?
1
vote
0answers
25 views

On gradient boosting and types of encodings

I am having a look at this material and I have found the following statement: For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly more ...
0
votes
0answers
13 views

Lightgbm - all the searched model have the same AUC

I'm trying to select the best model in a strongly imbalanced and large dataset using lightgbm. The target variable is binary. After performing a grid search, all the models have the same AUC. What ...
0
votes
0answers
9 views

Output value of a gradient boosting decision tree node that has just a single example in it

The general gradient boosting algorithm for tree-based classifiers is as follows: Input: training set ${\displaystyle \{(x_{i},y_{i})\}_{i=1}^{n},}{\displaystyle \{(x_{i},y_{i})\}_{i=1}^{n},}$a ...
0
votes
0answers
17 views

Understanding additive function approximation or Understanding matching pursuit

I am trying to read Greedy function approximation: A gradient boosting machine. On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach ...
0
votes
0answers
13 views

How to correctly weight observations in a decision tree

I'm building a boosting model and trying to fit a decision tree for the weak learner with a set of observation weights. I've seen two ways to do this: 1) bootstrap sample so that you have a higher ...
3
votes
1answer
142 views

How Do I Build a Quantile Regression Model with GradientBoostingRegressor from sklearn?

I am building a quantile regression model using scikit-learn's GradientBoostingRegressor algorithm. I was going to use GridSearchCV for hyperparameter optimization. Two questions: Does it make sense ...
0
votes
1answer
56 views

Gradient Boosting Positive/Negative feature importance in python

I am using gradient boosting to predict feature importance for a classification problem where one class is success and other is failed. However my model is only predicting feature importance for ...
5
votes
1answer
65 views

Understanding the weak learners in boosting

My understaning is that boosting is a method by which you have several weak models trained in sequence. Each is trained on the full trainig data, but with greater emphasis placed on the weaknesses of ...
0
votes
0answers
23 views

How are bagging and boosting methods of feature extraction/selection?

I am learning about boosting and bagging from the Amazon Web Services Machine Learning courses. In it, they describe bagging and boosting as ways to automate feature extraction and selection. My ...
0
votes
0answers
10 views

Summation in part 2d of Gradient Boost algorithm. Uniqueness of decision tree outputs for a given sample?

The below image shows the Gradient Tree Boosting algorithm from ESL. In part 2(d) of the algorithm, I am confused about the summation term. It's essentially the sum of the leaf values where a sample, $...
0
votes
1answer
87 views

Question about step size in gradient boosting

Above is the pseudocode for gradient boosting. In Step 2.3, we're computing a multiplier (or step length) $\gamma_m$. Suppose the loss function $L(y_i, \hat{y}_i) = \frac{1}{2}(y_i - \hat{y}_i)^2$. ...
1
vote
1answer
31 views

what does a high optimal shrinkage value indicate?

My optimal shrinkage value is high after comparing the MSE for different combinations of parameters. I'm wondering what does it mean to the data structure or signal structure? Can I say this dataset ...
0
votes
0answers
11 views

question about the paper “ADDITIVE LOGISTIC REGRESSION: A STATISTICAL VIEW OF BOOSTING”

I am reading the paper "additive logistic regression:a statistical view of boosting". I am confused about how to get equation (31),(32),(33) in the paper. Could anyone explain a bit please? ...

1
2 3 4 5
17