Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

How to calculate uncertainty for predictions coming from cascade of models?

I have developed a bunch of models to predict house prices. It is a 3 fold process: I fit a gbm (first_model) and get the first prediction (first_pred), there are some sub-models (simple lineer ...
0
votes
0answers
8 views

How can i increase the r2 value on validation data? [closed]

I'm having a problem finding a model for my regression problem, I've tried various models with no success. I'm using 5 fold cross validation and optimizing for the r2 metric, but I get results similar ...
0
votes
1answer
27 views

Likelihood that a prediction falls above (below) 110% (90%) of the prediction

For my client I have to predict some products' prices with gbm (scikit). So in the production, I am to give prediction intervals. That is, I need to provide how likely a real price falls above 110% or ...
0
votes
0answers
3 views

What are the benefits to multi-output objectives for tree-based boostings algorithms?

I see that there are implementations or calls for multiRMSE and multiclass objectives present in popular gradient boosting frameworks, including LightGBM and CatBoost. I can see practical benefits in ...
2
votes
0answers
51 views

Why fit on the gradient rather than minimize the loss directly when using Gradient Boosting?

In Friedman's paper on Gradient Boosting, he states the motivation for the gradient boosting algorithm is that it provides a framework of boosting for arbitrary loss functions. He then steps through a ...
0
votes
0answers
17 views

Predicting target variable for some a small group of a data by machine learning algorithms

This will be a general question about machine learning. Let say, I have such a data which have such variables: y(target variable): Salary x1: Age x2: Job x3: Location Let say, I want to predict y by ...
1
vote
1answer
33 views

how prediction of xgboost correspond to leaves values

I trained xgboost regressor. Now I want to see its fitted trees and to trace the prediction of an object. For simplicity I trained model with n_estimators = 1. But when I make a prediction for some ...
1
vote
0answers
34 views

Show that boosted decision stumps lead to an additive model? (ISLR question, proof attempted)

I am working through the Introduction to Statistical Learning questions in my spare time and gradually putting together a solutions set for my own learning. I've given this a solid attempt and but ...
0
votes
0answers
21 views

In gradient boost, do we still split nodes based on splitting criteria(impurity measure)?

Am I correct if i say that we use the loss function to calculate residuals, and the splitting criteria to determine which splits to make to predict these residuals? If this is the case how do we ...
1
vote
1answer
34 views

Interpretation of Gradient and Hessian for Categorical variables in Gradient Boost

Before searching for split points LightGBM sorts categories withing categorical features by: $\frac{\sum_{i=1}^{n} 1_{x_{i j}=x_{i k}} g_{i}}{\sum_{i=1}^{n} 1_{x_{i j}=x_{i k}} h_{i}}$ Where $g_i$ is ...
1
vote
1answer
28 views

XGBoost best tree path

I observed something that I could not make sense. Best tree inside the model uses same feature twice in a row. But the the thing I did not get is that: The first condition implies second condition. I ...
0
votes
0answers
14 views

Optimize learning rate of boosting algorithms efficiently with hyperopt

I've been exploring the hyperopt python library to tune parameters for boosting algorithms. Learning rate is one of the focal parameters for which efficient tuning appears more difficult. My ...
1
vote
1answer
20 views

Can missing values linked with outcome variable be a source of bias in machine learning

Background: In a binary classification problem (healthy vs. patients), I have a relatively small sample size (40 patients and 40 healthy subjects). I can include some additional subjects in both the ...
0
votes
0answers
27 views

Custom boosted pipeline in sklearn [closed]

I am interested in assembling a pipeline that has a series of models $[M_1, M_2, ..M_n]$ assembled in a boosted configuration. By that I mean: Fit $X$ on $Y$ with model $M_1$, get the residuals (call ...
0
votes
0answers
15 views

Decision Trees Outliers

How are Decision Trees affected with outliers both regression and classification ones? From my understanding, I see that in the classification context Decision Trees(DT) are robust to outliers as the ...
1
vote
1answer
30 views

How do you correctly use feature or permutation importance values for feature selection?

How do you properly use feature importance values from tree ensemble methods for feature selection without biasing your validation metric? I have a colleague at work that basically calculates ...
0
votes
0answers
16 views

Effect of adding extra feature created from other features at the data for gradient boosting algorithm?

Let say, I have such a data: y: 1 and 0 (target variable) x1: a1, a2 ,a3 values x2: b1,b2,.... bn values x3:......... x4:........... .... xk:...... where x1,x2,...., xk are the feature variables. We ...
0
votes
0answers
13 views

Unstable Out-of-Sample Prediction with Gradient Boosting Trees

I have a continuous response, ranging between -100 and 100, but it's highly leptokurtic at 0. I also have a lot of predictors to use. After variable reduction and parameter tuning, the prediction of ...
0
votes
0answers
54 views

Model Selection, Tree-Based Algorithms, Multiple Comparisons, and Hypothesis Testing

Only data scientist in an organization and I could really use a sounding board here. In Phase One of a project I deployed four models and served their average as the prediction. I used Random Forest ...
0
votes
0answers
20 views

Is number of features greater than minority class size a problem for tree based models (boosting, random forest)?

In Logistic Regression there is the "One in Ten Rule" (https://en.wikipedia.org/wiki/One_in_ten_rule). For example, there is a sample of 2000 customers, and 50 of them belong to the positive ...
0
votes
0answers
33 views

Custom Loss Function in Decision Tree for Ranking

I have built a Decision Tree and Adaboost model from scratch in Python and am now trying to customize the loss function being used. I am hoping to use a ranking loss function but am having troubles ...
1
vote
1answer
27 views

Does majority-vote boost weak learners to strong learners?

A learner is a function mapping finite vectors with elements in $X\times\{0, 1\}$ onto binary functions on $X$. Given a set $H$ of binary functions on $X$, we say that: A learner $(\delta, \epsilon)$-...
2
votes
0answers
18 views

Gradient boosting: tree that fits the gradient of the custom loss function always uses squared loss?

With gradient boosting for regression, there are 2 loss functions, i.e: a custom loss function that we calculate the gradient for: $L(y_i,\hat{y_i})$ the loss function used by the tree that fits the ...
1
vote
0answers
28 views

Gradient boosting doubts

I am new to the concept of Gradient Boosting and i have a few doubts related to it. It will be helpful if some one can explain them. 1) Gradient boosting is gradient descent in functional space As ...
0
votes
0answers
13 views

Issues with tree based learners for Boosting Machines or RFs [duplicate]

I'm wondering if tree based methods are capable of making predictions that are larger in magnitude than the largest training observations? Given my understanding of decision trees and partitioning ...
0
votes
1answer
18 views

Can we build decision trees faster if we use an approximate local maximum search method to find split points?

In CART and similar algorithms, when we want to create a split, we iterate over all possible split values, evaluate the score (Gini, information gain) produced by the split. Then we pick the split ...
0
votes
0answers
12 views

Why does likelihood-based boosting use a penalization instead of the small-step-length approach

In statistical gradient boosting, in every iteration the additive predictor is only updated by a small proportion of the value of estimated base-learner coefficient that would actually best fit the ...
0
votes
0answers
12 views

H2O splitting (node improvement) method

H20 says in the documentation (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/gbm-faq/splitting.html) that splitting on a feature for regression gbms is based on the reduction in squared ...
1
vote
1answer
15 views

lightgbm model diagnostics

I am currently building a house price predictor. My lightgbm errors look like the one the one below (illustrative). It shows that there is a pattern in my errors. Can someone explain how to resolve ...
0
votes
0answers
18 views

why do i get different scales in my feature importance reports for xgboost and light gradient boosting?

i ran the exact same process to compare models (xgboost and light gradient boosting) however i get very different feature importance report scales. e.g. for xgboost the scale in the report is from 0 - ...
0
votes
0answers
14 views

Data Considerations for GBM

I am using data summarized in SQL and having a continuous target variable with the weights= option being used since, well, the data is summarized. And each variable combination (just over 15,000) has ...
1
vote
1answer
12 views

How is first tree in Boosting constructed

How is the first tree in GBM constructed, and how the node splitting criteria for the first tree is decided. Can someone please explain, what we are predicting for the first tree (even assuming a ...
2
votes
1answer
131 views

In XGboost are weights estimated for each sample and then averaged

The weights in XGBoost are determined by gradient boosting. So, each sample gets a weight and as each leaf has multiple samples, initially each leaf has multiple weights. But, as a single weight is ...
0
votes
0answers
30 views

Can I build deviance residuals from an XGBoost model that learns an exponential family parameter?

I'm taking a course on GLMs after a few years of using machine learning models. The good about GLMs is how the probabilistic model ties in with the estimation and evaluation. So I'm trying to transfer ...
0
votes
0answers
30 views

I Have A Doubt Regarding AdaBoost Weight Update Rule As Various Sources Cite Two Different Things

I was reading about AdaBoost from Hands-On Machine Learning with Scikit-Learn and TensorFlow and came across the formula used for updating the weights. In the book, it was the following: But in this ...
0
votes
0answers
15 views

What does the loss function adjust in gradient boosting

I get that a gradient boosting algorithm adds weak learners together in order to fit some data. I also get that each subsequent learner after the first one fits the residuals of the current model. ...
0
votes
0answers
37 views

KL Divergence Between Ground Truth and Prediction

I've got four (non-linear, tree-based) models in production and using the average of them as the served prediction. We get ground truth data immediately. During training the optimized candidate models ...
0
votes
2answers
34 views

Feature importance changes drastically when adding other features

I have a model (GBDT) where adding a feature X is not important (according to SHAP), but when I add other features, and add X again, now feature X is the second most important! What could explain that?...
1
vote
1answer
140 views

Partial dependence plot, GBM multinomial

I'm using gbm package to create a gradient boosting model with multinomial family. I have problems in plotting and interpreting partial dependence plots with this type of models. an example: ...
0
votes
1answer
34 views

What is the impact of a dummy variables to boosted trees?

I am currently reading the book "Random Forests" by Yu. L. Pavlov. Then it came across my mind the question If I were to use ensembled tree, say XGBOOST, do I need to transform each ...
3
votes
1answer
118 views

How Gradient Descent is used for classification with Decision Trees?

I'm not able to see how do we use gradient descent to minimize the loss of binary classification with decision tree. What I understood is that we first have a model (decision tree) that try to predict ...
0
votes
0answers
20 views

Is my understanding and presentation of concept of Gradient Boosting correct?

Initially the model is trained with a training set $\{x_{i}, y_{i}\}_{i=1}^{n}$ by minimizing a differentiable loss function $L(y, F(x))$, and, is initialized with a constant value, \begin{align*} F_{...
0
votes
0answers
18 views

Different outcomes between XGBoost and GBM observation weights

I've noticed that the weights argument in the xgboost and gbm r packages don't have the same effect. I had hoped to move from gbm to xgboost for performance reasons, but am not sure how to do so given ...
2
votes
1answer
74 views

Zero inflated Count Data treatment with XGBOOST

I am planning to run an xgboost in response data that is: Count data (0 to 15) Very right skewed Zero inflated (lots more zero than other counts) In the XBG package with R, I have specified count:...
1
vote
1answer
85 views

Is a high learning rate irrelevant when dropping the first or last tree in a GBDT with 100 trees?

Suppose we've trained a GBDT model with 100 trees with a fairly high learning rate. Consider two cases: We drop the first tree in the model We drop the last tree in the model We then compare models ...
0
votes
0answers
25 views

What is this symbol in Adaboost algorithm (SAMME)?

According to the original paper of SAMME algorithm (Adaboost for multiclass problems), it is described as follows: The general idea is very straightfoward, except for this symbol: The author didn't ...
2
votes
1answer
70 views

Overfit in aggregated models: boosting versus simple bagging

Let's fix a bagging setup, where several models are build independently and than somehow aggregated. It is intuitive that increasing the number of weak learners ( N ) does not lead to overfit ( in the ...
0
votes
0answers
20 views

Question on Model Validation and Interpretation

I’m a beginner here who is trying to build a model for my personal project (albeit using data I got from my firm). The data is a call center dataset , and Im trying to predict the number of successful ...
1
vote
1answer
74 views

linear weak learners for Xgboost

I see now that Xgboost documentation only considers trees as weak learners, but I remember well tath linear models were an option too, I wander if they are still supported. Anyway, I always assumed ...
0
votes
1answer
25 views

In Gradient Boosting over Decision Trees what does it mean to reconstruct the residual

So I am trying to understand what happens in GBDT and particularly I want to know what it mean to "reconstruct the residual." The way I understand it is that the next tree will use the ...

1
2 3 4 5
18