Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

1
vote
0answers
16 views

On what types of datasets do tree-based models not do well?

Are there examples where splitting on the best feature/threshold combination is not actually the best way to split the tree, and that better results could be got by choosing a different feature but ...
1
vote
0answers
6 views

Boosted regression trees - clarification of algorithm step

In my statistic learning textbook, there is an algorithm, "Boosting for Regression Trees". As step 1 in the algorithm, it is said to set $\hat{f}(x) = 0$ and $r_i = y_i$ for all $i$ in the training ...
1
vote
0answers
11 views

Decision tree / boosting tree in Tensorflow?

When I was looking for some way to break the bottleneck of memory limitation in xgboost, I found there are some boosting tree algorithm that is implemented in Tensorflow such as here. I have some ...
3
votes
0answers
21 views

Is Gradiant Boosting a generalization of Adaboost?

I read somewhere that Gradiant boosting is a generalization of Adaboost. However, I cannot see why. Can Anyone elaborate?
1
vote
0answers
32 views

Machine Learning on Extremely Low Signal Data

I have terabytes of data with an extremely low signal to noise ratio, with the following characteristics: The relationship between the features and the response variable can change over time I'm ...
0
votes
1answer
17 views

How to correctly retrain model using all data, after cross-validation with early stopping

I have a classification task that doesn't have loads and loads of data, so I'd like to make the most of the data. I have a boosting model and I've performed 5-fold CV, using the validation fold for ...
1
vote
2answers
59 views

Which formula for GBM is correct?

I am trying to write a simple GBM simulator. Unfortunately, the task has turned rather difficult. The first approach I looked into was the most obvious. I could use the analytic solution for the GBM ...
1
vote
0answers
29 views

XGBoost Poisson Objective Function When Data is Over-dispersed [closed]

I am modeling very over-dispersed count data with the goal of prediction. The data is not zero inflated (there are no zeros), but there are a lot of values of 1. ...
0
votes
0answers
14 views

What is the interpretation of GBM model output via caret train function?

I am running GBM method for a Classification problem in Caret R. With Verbose=T and Resampling method as "repeatedcv" with 5-folds & 5-repeats, I get the following values printed in console: ...
4
votes
1answer
53 views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
3
votes
0answers
75 views

About Partial dependence for Poisson GLM

Can someone tell me what would be the expression for calculating the partial dependence on a GLM model with family specified as Poisson? From applying Friedman partial dependence estimation ...
0
votes
0answers
20 views

Data standardization or normalization in GBDT [duplicate]

Is it necessary to do data normalization(standardization) before using gbdt?what effect does it have if I don't do that proprecessing?
2
votes
1answer
40 views

How to constrain gradient boosting predictions to be non-negative?

I have a regression problem with the target always nonnegative, I used gbdt as the model, but sometimes the model outputs negative prediction value.Is there any way to output nonnegative value using ...
0
votes
0answers
10 views

Computational Optimization of Cross-Validation in Training Models

TL;DR: If I run a 5-fold cross-validation which assigns 1 fold per core when my CPU only has 4 cores, is the 5th as costly (with respect to computation time) as the first 4 folds? Context: I am ...
1
vote
1answer
15 views

Decision Tree - What to avoid first a Skewed dataset or reduce too much the number of bins

So I have a dataset with a categorical column rather skewed. Lets imagine something like this: ...
1
vote
0answers
19 views

How do Gradient-boosted trees cope with new categories after training?

I'm looking at using a Gradient-boosted tree model to predict categories for a dataset. The data has has multiple categorical variables which have high cardinality. I've converted the dataset into ...
2
votes
1answer
34 views

Get the number of weak learner - ebmc package of R - implementing class imbalance RusBoost on my dataset

I'm new to class imbalance and applying class imbalance technique 'RusBoost' on my dataset. I'm using ebmc package from R. I'm having difficulties to get its arguements values, as per the ...
0
votes
0answers
28 views

Gradient Boosted Regression - decide number of trees?

By adding arbitrarily many trees, seems like the $R^2$ value can be as close to 1.0 as we want. This doesn't seem correct. How do we determine the optimal number of trees? Should I use a form of ...
1
vote
1answer
27 views

Find a weak learner in Boosting

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, PDF source Question 1: Why find the weak learner by the formula ...
0
votes
0answers
26 views

How to improve GBM performance

I'm trying to model insurance losses, with a Tweedie distribution. I have a data set of about 40 million records, and over 100 independent variables. My response variable is "loss", I take the log of ...
0
votes
0answers
16 views

Residuals' spatial autocorrelation in Boosted Regression Trees after correcting for it

I am running boosted regression trees (BRT) in R, with the package dismo and I have included a predictor (residual autocovariate) that, in theory, correct for spatial autocorrelation, following a ...
1
vote
1answer
21 views

How does gbm for classification work?

I have a got a fair idea about how it works in regression where each successive decision tree tries to predict the residual (negative gradient for loss function) and the predicted value gets added to ...
1
vote
0answers
31 views

problem with Gradient Boosting regression predicting negative values

i have data set of which the target values are all positives but when apply Gradient Boosting regression i get some negative predictions but also a good $R^2$ score ($R^2$ = 0.823) but the more ...
0
votes
0answers
23 views

What happens when the feature importance plot is dominated by only one feature?

I got a feature importance plot from my gbm model, where one of the feature shows a very high value of feature importance as compared to the other variables. Will that be affecting my predictions in a ...
1
vote
2answers
146 views

Why does XGBoost have a learning rate?

Having used XGBoost a fair bit, clearly changing the learning rate dramatically affects the algorithm's performance. That said, I really can't understand the theoretical justification for it. It makes ...
0
votes
0answers
28 views

Weights vs offsets in logistic tree models

I'm interested in the differences in interpretation and functionality regarding weights and offsets in logistic regression trees. In my case, I am using XGBoost trees for logistic regression where the ...
3
votes
0answers
39 views

Gradient Boosting: When to stop doing manual feature engineering?

As an example Let's assume we do have feature-columns $X_1$ and $X_2$ and $X_3$ and target $Y$, and it just so happens that $Y$ is the noisy spearman correlation of the three features. Would it help ...
2
votes
1answer
86 views

What are the implications of scaling the features to xgboost?

Doing research about the xgboost algorithm I went through the documentation. I have heard that xgboost does not care much about the scale of the input features In this approach trees are ...
0
votes
0answers
58 views

What are the final predictions in tree based models?

I always thought that there are only two ways the predictions from the final leaf are extracted in a tree based model: For regression problems take the average of the continuous variable to be ...
3
votes
1answer
83 views

What is Oblivious Decision Tree and Why?

I read the catboost paper and they mentioned the trees are oblivious decision tree. What is the definition of oblivious decision tree? I found two possible candidates but not sure if they are the ...
0
votes
1answer
38 views

Cluster analysis with boosting models for better predictions?

let's say that we have a simple, binary classification problem (with many predictors and many observations) and want to fit for example some kind of boosting algorithm to obtain resutls. Let's also ...
1
vote
0answers
19 views

What stops an additional layer from destructively adding to the previous layers in an Adaboost?

Algorithmic representation of Discrete Adaboost is: Start with weights $w_i = 1/N, i = 1, .. N$ Repeat for $m = 1,...,M$ Fit the classifier $f_m(x) \in \{-1,1\}$ using weights $w_i$ on the training ...
0
votes
0answers
30 views

Combining Gradient boosted trees after multiple imputation

Currently I am working with a gradient boosted tree model fit onto a multiple imputed dataset. For those who don't know multiple imputation: It predicts missing values and imputes that value with ...
1
vote
0answers
102 views

Understanding regularization in xgboost

A general loss function is: \begin{split}\text{obj} = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t)}) + \sum_{i=1}^t\Omega(f_i) \\ \end{split} which is prediction cost + regularization cost A decision tree is ...
1
vote
1answer
36 views

Training binary classifiers with huge dataset with mostly negative examples [duplicate]

I would like to build an ensemble classifier (possibly boosting) on a huge training dataset (>> 1e7 examples) where the proportion of positive examples is around 5%. And what I am interested in are ...
0
votes
0answers
15 views

General guidance on selecting the modelling appraoch

I need some general guidance on choosing different modelling approaches out there. I am mainly investigation the effect of climate variables on crop yield and develop a model to predict yield based ...
0
votes
0answers
30 views

Boosting - learners with weight?

I'm trying to understand the Boosting method and I'm very confused by what I see on the internet as there are different explanations and I'm not sure some of them are true. Here's what I've ...
0
votes
1answer
30 views

Writing the equation of boosted regression trees model [closed]

This is the formula for fitting a boosted regression trees from gbm package: ...
2
votes
0answers
35 views

Handling missing data in gbm R

I was wondering on some more information on how gbm handles missing data following this tutorial. http://s3l.stanford.edu/blog/?p=73 Decision trees are robust to outliers and handle missing data ...
0
votes
0answers
32 views

Tree complexity in boosted regression trees [duplicate]

In boosting regress trees, there are many tuning parameters and I am interested in the tree complexity also called the interaction depth while using the R package GBM: I want to understand more ...
2
votes
0answers
44 views

How to work with unbalanced datasets in generalized Boosted Regression Models using gbm in r?

I'm using generalized boosted regression models to explore what is the contribution of 20 independent environmental variables (x1, x2, ...., x20) to the explanation of the variability of the dependent ...
1
vote
1answer
53 views

Using trees for conditioned linear regression

I want to explore using tree methods to condition a linear regression. Let's say the baseline regression is of the form $y = \beta x + \epsilon$. In addition, I have conditioning data, a vector $c \...
5
votes
1answer
334 views

How does gradient boosting calculate probability estimates?

I have been trying to understand gradient boosting reading various blogs, websites and trying to find my answer by looking through for example the XGBoost source code. However, I cannot seem to find ...
0
votes
0answers
37 views

what is instance weight in boosting?

Hi, I am reading something about Boosting, and I had hard time understanding one of the steps in boosting - assign greater weights to those instances. What does the sentence - assign greater weights ...
1
vote
1answer
47 views

Should `predict_proba` average be population average?

I'm running a gradient boosted tree in sklearn and running on some test data. The frequency of positive examples ('1') in the test data should be around 10%, which ...
0
votes
0answers
17 views

Testing GBM regression model

I understand testing a GBM on classification problems is quite easy, just get the confusion matrix and calculate all the metrics. However, how do i test a GBM for regression models? Where the output ...
1
vote
0answers
18 views

Use GBM to find the optimum configuration of variables given certain conditions

I have a data set containing historical data of a SAG Ball Mill There are two types of variables: Mineral Characteristics, and Mill Paremeters. Mineral Characteristics are given and can not be ...
2
votes
0answers
125 views

What does L2-regularization in LightGBM do?

I use LightGBM for regression task and I'm planning to use L2-regularization to avoid overfitting. It's quite clear for me what L2-regularization does in linear regression but I couldn't find any ...
1
vote
0answers
29 views

Is Gradient Boosted Tree boosting on the residuals or on the complete training set?

It is not fully clear to me if Gradient Boosted Trees are boosting on the residuals of each stage or on the complete training set? From the Wikipedia page it seems to me that the Overal "Algorithm" ...
1
vote
2answers
47 views

What is the relationship between gradient boosting and gradient decent? [duplicate]

What is the relationship between gradient boosting and gradient decent? Are the closely related and how are they related to each other?