Boosting is a process of finding & combining weakly predictive models into a strongly predictive model.

learn more… | top users | synonyms

0
votes
0answers
8 views

Comparing gradient boost trees implementations

Any thoughts on which implementation of gradient boosted trees is more accurate and faster? opencv xgboost http://zhanpengfang.github.io/418home.html Thanks.
0
votes
1answer
23 views

How are the outcomes that generated from different predictive models combined to get more accurate predictions?

The simple average is commonly used to combine the predictions of different predictive models. Apart form the simple average, what are the other methods that can be used for combining the predictive ...
0
votes
1answer
28 views

Feature importance in gradient boosted trees

I am tuning the parameters of a gradient boosting regression tree algorithm and find it hard to understand the importance of some variables. Here is the case.. when the number of estimators is ...
0
votes
1answer
132 views

Gradient for logistic loss function

I would ask a question related to this one. I found an example of writing custom loss function for xgboost here: ...
0
votes
0answers
4 views

Cross-validation on XGBClassifier for multiclass classification in python [migrated]

I'm trying to perform cross-validation on a XGBClassifier for a multi-class classification problem using the following code adapted from http://www.analyticsvidhya.com/blog/2016/03/complete-guide-...
0
votes
0answers
9 views

what 's the difference between GBM and BRT [duplicate]

Gradient Boosting Machine (GBM) Boosted Regression tree (BRT) what's the difference? It seems the same.
1
vote
0answers
43 views

Why do boosting overfit on the data with uniform noise?

i read about it but i didn't get the idea, and actually i didn't find many pages that talk about uniform noise with boosting, is it rare to happen or what? another question: i read in some pages that ...
0
votes
1answer
39 views

preprocessing and input format for gradient boosted trees

I'm using MlLib library de Spark and trying to perform Gradient-Boosted Trees algorithm on my data, that has mostly categorical features (and just two numerical features). in the example given in ...
0
votes
1answer
42 views

Improving a boosted regression model or change?

I am looking at a data set that contains multiple predictors and a continuous response. Using dismo along with gbm I built (a terrible one?) model. Using the package sROC, I got an AUC or 0.48 - so my ...
0
votes
0answers
87 views

XGBoost - Can we find a “better” objective function than RMSE for regression?

If we think back to linear models for a moment, we have Ordinary Least Squares (OLS) versus Generalized Linear Models (GLM). Without going too in-depth, it can be said that GLMs "improve" upon OLS by ...
0
votes
1answer
59 views

Understanding results of xgboost, parameter tuning [closed]

I ran xgboost with below parameter setting: ...
8
votes
4answers
547 views

Gradient boosting machine accuracy decreases as number of iterations increases

I'm experimenting with the gradient boosting machine algorithm via the caret package in R. Using a small college admissions dataset, I ran the following code: <...
0
votes
0answers
9 views

How should I define qoffset for QuantReg family in mboost?

The QuantReg family of the mboost package has two parameters: tau (the target quantile) and <...
1
vote
0answers
30 views

How to obtain a confidence interval or a measure of prediction dispersion when using xgboost for classification?

How to obtain a confidence interval or a measure of prediction dispersion when using xgboost for classification? So for example, if xgboost predicts a probability of an event is 0.9, how can the ...
0
votes
1answer
27 views

xgboost parameters - how can we use max.depth parameter with binary:logistic objective

I'm new to xgboost package and here is the doc on the parameters of this library for your reference. My question is, logistic regression doesn't do binary splitting and build a tree unlike decision ...
0
votes
0answers
23 views

Multinomial loss derivative in gradient boost

I'm struggling hard to understand the derivative Jerome Friedman uses to extend gradient boosting to the multiclass case using multinomial logistic loss in his paper on Gradient Boosting: Greedy ...
0
votes
0answers
8 views

Is there any good way to blend strong model?

I want to blend some boosting model like gradient boosting tree, random forest or something also(And I am using them for regression). These model are all strong model, and it seems strange to using ...
1
vote
2answers
34 views

Boosted Trees classification

I'm using R's gbm() package to do a boosted classification problem, where my response variable is a binary variable taking values of 0 and 1. I have 11 predictors in my data set. After running the gbm(...
0
votes
0answers
19 views

xboost performance with predicted values as input

I have predicted probability of loss using different features. Now when I used this with non-important feature to predict probability of loss. It is very close first one. logloss was close to 0.11. ...
1
vote
1answer
50 views

Why are gradient boosting regression trees good candidates for ranking problems?

I have been reading up on gradient boosting machines, and in particular GBRT's. I've come across numerous mentions (and finally tracked down some papers) on applying these models to ranking problems - ...
1
vote
0answers
99 views

Generate code for sklearn's GradientBoostingClassifier

I want to generate code (Python for now, but ultimately C) from a trained gradient boosted classifier (from sklearn). As far as I understand it, the model takes an initial predictor, and then adds ...
0
votes
0answers
7 views

Gbm.plot y axis

I am fitting a boosted regression tree to count data. The response is distributed Poisson. When I plot the model's partial residuals using gbm.plot, the y-axis goes from -1 to 1. Are these plots ...
0
votes
0answers
23 views

Is boosting resistant to overfitting for both number of iterations and number of features?

Boosting methods (such as the popular xgboost) do not tend to overfit when we use many iterations - Schapire and Freund. Are they also resistant to overfitting when ...
0
votes
1answer
25 views

Do Random Forests use boosting

Ok so I think I have listened to a few wrong discussions on random forests because now I have a very confused question. With respect to Random Forests and bagging/bootstrapping, I'm good there. The ...
1
vote
0answers
52 views

Tuning Parameters for Boosting/Bagging/Random Forest

I want to use tree-based classifiers for my classifiaction problem. I'm thinking about bagging, boosting (AdaBoost, LogitBoost, RUSBoost) and Random Forest but I'm unsure about the tuning parameters, ...
0
votes
0answers
27 views

Implementing an Adaboost Classifier

I have generated an adaboost classifier in Weka on a dataset where each instance falls into one of two classes. The result was a number of decision trees, each assigned a weight. What is the ...
0
votes
0answers
75 views

eval_set on XGBClassifier

can someone explain what does the eval_set parameter do on the XGBClassifier? I thought that by using eval_set, the algorithm would do some sort of grid search and find the best model to fit on train ...
0
votes
0answers
54 views

Understanding the approach behind variable importance returned with Xgboost method in R package caret

I recently implemented the R package caret, for a binary categorical outcome regarding a transcriptomic microarray dataset. As i used the method from the xgboost package(method="xgbtree"), then i used ...
1
vote
0answers
29 views

Is there a minimum event rate required for the gradient boosting to work?

I am trying to run gradient boosting in enterprise miner on a dataset which has event rate of about 2% and sample size is about 1m. It fails to produce any output. Which makes me think, is there a ...
0
votes
1answer
42 views

(Boosted) regression trees versus model trees - rule of thumb what to use when

I apply (boosted) regression trees to build predicitive models with continuous outcome (xgboost and gbm). While regression trees ...
3
votes
1answer
85 views

Additive bias in xgboost (and its correction?)

I am taking part in a competition right now. I know it is my job to do that well, but maybe somebody wants to discuss my problem and its solution here as this could be helfull for others in their ...
1
vote
2answers
57 views

How to interpret when a variable is not significant in logistic regression while having the highest variable importance in a tree-based model [duplicate]

I'm building a binary classifier with logistic regression and boosting. Just like the case that I described in the title, I am a little bit confuse on how to explain the result of those two models ...
0
votes
0answers
11 views

Counting parameters of a gradient boosted decision tree

Given the number of predictors and the depth of the trees, how many are the parameters of the models in a boosted decision tree? Is there a simple formula to count all the parameters of the model as ...
-1
votes
1answer
154 views

How does XGboost (Python) differentiate between a nominal variable and a continuous variable?

Assume the data in one dimension is (-1.0, 2.0, 2.5, 3.0, 5.0). Does XGboost regard it as a nominal or a continuous variable?
1
vote
1answer
182 views

Discussion about overfit in xgboost

My set-up is the following: I am following the guidlines in "Applied Predictive Modelling". Thus I have filtered correlated features and end up with the following: 4900 data points in the training ...
4
votes
2answers
141 views

Classification with Gradient Boosting : How to keep the prediction in [0,1]

The question I am struggling to understand how the prediction is kept within the $[0,1]$ interval when doing binary classification with Gradient Boosting. Assume we are working on a binary ...
0
votes
1answer
31 views

Gradient Boosting: Is it possible to use a weak classifier?

My understanding is that a regressor has to be used to fit to the residual. Is it possible to directly apply a classifier? If so, what are the requirements/restrictions?
0
votes
0answers
18 views

does xgboost's eval_metric changes the loss function being optimized?

I'm using xgboost with the reg:logistic objective. As far as I understand, that means that I'm trying to optimize the log-...
1
vote
1answer
91 views

Feature subsampling with gradient boosting

A key component in building random forest models is feature subsampling, i.e., building each individual tree with only a percentage of predictors chosen randomly by tree. The literature often ...
2
votes
0answers
98 views

Bagging of xgboost

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to ...
3
votes
0answers
26 views

Data reduction and xgboost(or other boosting and decisision tree methods)

I wonder, does data reduction(ex:factor analysis) have an impact on the result of boosting(ex:xgboost) or decision trees methods other than time gain?
-1
votes
1answer
123 views

Missing value in categorical data with xgboost

I have a dataset with many binary indicators and five categorical variables, sex, city, building, precinct of stop and race. I'm going to use gradient boosting methods, but come up with the problem ...
1
vote
1answer
137 views

XGBoost (Extreme Gradient Boosting) or Elastic Net More Robust to Outliers

I have recently been doing work with predictive models for a continuous response. I am doing a comparison between Elastic Net (glmnet) package in R and XGBoost (<...
6
votes
1answer
229 views

Loss function Approximation With Taylor Expansion

As an example, take the objective function of the XGBOOST model on the $t$'th iteration: $$\mathcal{L}^{(t)}=\sum_{i=1}^n\ell(y_i,\hat{y}_i^{(t-1)}+f_t(\mathbf{x}_i))+\Omega(f_t)$$ where $\ell$ is ...
2
votes
1answer
83 views

Standardizing numerical and encoding of categorical data for training boosted decision tree

Is there a "best practice" way of standardizing numerical and encoding of categorical data for training boosted decision tree? Both for classification and regression problems
1
vote
0answers
77 views

How to tune the weak learner in boosted algorithms

It is commonly said that boosted algorithms (adaboost, gradient boosted trees) are composed of many "weak" learners. Let's stick to decision trees as the base learners. Some empirical studies ...
1
vote
1answer
237 views

xgboost - what is the difference between the tree booster and the linear booster?

I am aware of gradient boosted trees. The extreme-gradient boosting algrithm is widely applied these days. What excactly is the difference between the tree booster (gbtree) and the linear booster (...
1
vote
0answers
28 views

Model stacking, should the folds in the training set be the same?

I am stacking various models (Gradient Boosting Machines, Random Forests, Linear Regressions) using a k-fold cross validation for the train set $X_{train}$, therefore obtaining out-of-sample ...
0
votes
0answers
112 views

Is gradient boosting appropriate for data with low event rates like 1%?

I am trying gradient boosting on a dataset with event rate about 1% using Enterprise miner, but it is failing to produce any output. My question is, since it a decision tree based approach, is it even ...
1
vote
0answers
39 views

Feature binarization for RF/GBMs?

Are there any advantages to feature binarization for random forests or gradient-boosted machines? For example, suppose I am predicting snowstorms for the next day using various past measurements - ...