Boosting is a process of finding & combining weakly predictive models into a strongly predictive model.

learn more… | top users | synonyms

0
votes
0answers
9 views

xboost performance with predicted values as input

I have predicted probability of loss using different features. Now when I used this with non-important feature to predict probability of loss. It is very close first one. logloss was close to 0.11. ...
1
vote
1answer
25 views

Why are gradient boosting regression trees good candidates for ranking problems?

I have been reading up on gradient boosting machines, and in particular GBRT's. I've come across numerous mentions (and finally tracked down some papers) on applying these models to ranking problems - ...
1
vote
0answers
73 views

Generate code for sklearn's GradientBoostingClassifier

I want to generate code (Python for now, but ultimately C) from a trained gradient boosted classifier (from sklearn). As far as I understand it, the model takes an initial predictor, and then adds ...
0
votes
0answers
5 views

Gbm.plot y axis

I am fitting a boosted regression tree to count data. The response is distributed Poisson. When I plot the model's partial residuals using gbm.plot, the y-axis goes from -1 to 1. Are these plots ...
0
votes
0answers
16 views

Is boosting resistant to overfitting for both number of iterations and number of features?

Boosting methods (such as the popular xgboost) do not tend to overfit when we use many iterations - Schapire and Freund. Are they also resistant to overfitting when ...
0
votes
1answer
20 views

Do Random Forests use boosting

Ok so I think I have listened to a few wrong discussions on random forests because now I have a very confused question. With respect to Random Forests and bagging/bootstrapping, I'm good there. The ...
1
vote
0answers
27 views

Tuning Parameters for Boosting/Bagging/Random Forest

I want to use tree-based classifiers for my classifiaction problem. I'm thinking about bagging, boosting (AdaBoost, LogitBoost, RUSBoost) and Random Forest but I'm unsure about the tuning parameters, ...
0
votes
0answers
18 views

Implementing an Adaboost Classifier

I have generated an adaboost classifier in Weka on a dataset where each instance falls into one of two classes. The result was a number of decision trees, each assigned a weight. What is the ...
0
votes
0answers
30 views

eval_set on XGBClassifier

can someone explain what does the eval_set parameter do on the XGBClassifier? I thought that by using eval_set, the algorithm would do some sort of grid search and find the best model to fit on train ...
0
votes
0answers
29 views

Understanding the approach behind variable importance returned with Xgboost method in R package caret

I recently implemented the R package caret, for a binary categorical outcome regarding a transcriptomic microarray dataset. As i used the method from the xgboost package(method="xgbtree"), then i used ...
1
vote
0answers
20 views

Is there a minimum event rate required for the gradient boosting to work?

I am trying to run gradient boosting in enterprise miner on a dataset which has event rate of about 2% and sample size is about 1m. It fails to produce any output. Which makes me think, is there a ...
0
votes
1answer
28 views

(Boosted) regression trees versus model trees - rule of thumb what to use when

I apply (boosted) regression trees to build predicitive models with continuous outcome (xgboost and gbm). While regression trees ...
3
votes
1answer
58 views

Additive bias in xgboost (and its correction?)

I am taking part in a competition right now. I know it is my job to do that well, but maybe somebody wants to discuss my problem and its solution here as this could be helfull for others in their ...
1
vote
2answers
47 views

How to interpret when a variable is not significant in logistic regression while having the highest variable importance in a tree-based model [duplicate]

I'm building a binary classifier with logistic regression and boosting. Just like the case that I described in the title, I am a little bit confuse on how to explain the result of those two models ...
0
votes
0answers
10 views

Counting parameters of a gradient boosted decision tree

Given the number of predictors and the depth of the trees, how many are the parameters of the models in a boosted decision tree? Is there a simple formula to count all the parameters of the model as ...
-1
votes
1answer
53 views

How does XGboost (Python) differentiate between a nominal variable and a continuous variable?

Assume the data in one dimension is (-1.0, 2.0, 2.5, 3.0, 5.0). Does XGboost regard it as a nominal or a continuous variable?
1
vote
1answer
84 views

Discussion about overfit in xgboost

My set-up is the following: I am following the guidlines in "Applied Predictive Modelling". Thus I have filtered correlated features and end up with the following: 4900 data points in the training ...
4
votes
2answers
127 views

Classification with Gradient Boosting : How to keep the prediction in [0,1]

The question I am struggling to understand how the prediction is kept within the $[0,1]$ interval when doing binary classification with Gradient Boosting. Assume we are working on a binary ...
0
votes
1answer
29 views

Gradient Boosting: Is it possible to use a weak classifier?

My understanding is that a regressor has to be used to fit to the residual. Is it possible to directly apply a classifier? If so, what are the requirements/restrictions?
0
votes
0answers
12 views

does xgboost's eval_metric changes the loss function being optimized?

I'm using xgboost with the reg:logistic objective. As far as I understand, that means that I'm trying to optimize the ...
1
vote
1answer
51 views

Feature subsampling with gradient boosting

A key component in building random forest models is feature subsampling, i.e., building each individual tree with only a percentage of predictors chosen randomly by tree. The literature often ...
2
votes
0answers
58 views

Bagging of xgboost

The extreme-gradient boosting algorithm seems to be widely applied these days. I often have the feeling that boosted models tend to overfit. I know that there are parameters in the algorithm to ...
3
votes
0answers
19 views

Data reduction and xgboost(or other boosting and decisision tree methods)

I wonder, does data reduction(ex:factor analysis) have an impact on the result of boosting(ex:xgboost) or decision trees methods other than time gain?
-1
votes
1answer
49 views

Missing value in categorical data with xgboost

I have a dataset with many binary indicators and five categorical variables, sex, city, building, precinct of stop and race. I'm going to use gradient boosting methods, but come up with the problem ...
1
vote
1answer
69 views

XGBoost (Extreme Gradient Boosting) or Elastic Net More Robust to Outliers

I have recently been doing work with predictive models for a continuous response. I am doing a comparison between Elastic Net (glmnet) package in R and XGBoost ...
6
votes
1answer
148 views

Loss function Approximation With Taylor Expansion

As an example, take the objective function of the XGBOOST model on the $t$'th iteration: $$\mathcal{L}^{(t)}=\sum_{i=1}^n\ell(y_i,\hat{y}_i^{(t-1)}+f_t(\mathbf{x}_i))+\Omega(f_t)$$ where $\ell$ is ...
2
votes
1answer
39 views

Standardizing numerical and encoding of categorical data for training boosted decision tree

Is there a "best practice" way of standardizing numerical and encoding of categorical data for training boosted decision tree? Both for classification and regression problems
1
vote
0answers
47 views

How to tune the weak learner in boosted algorithms

It is commonly said that boosted algorithms (adaboost, gradient boosted trees) are composed of many "weak" learners. Let's stick to decision trees as the base learners. Some empirical studies ...
1
vote
1answer
97 views

xgboost - what is the difference between the tree booster and the linear booster?

I am aware of gradient boosted trees. The extreme-gradient boosting algrithm is widely applied these days. What excactly is the difference between the tree booster (gbtree) and the linear booster ...
1
vote
0answers
26 views

Model stacking, should the folds in the training set be the same?

I am stacking various models (Gradient Boosting Machines, Random Forests, Linear Regressions) using a k-fold cross validation for the train set $X_{train}$, therefore obtaining out-of-sample ...
0
votes
0answers
52 views

Is gradient boosting appropriate for data with low event rates like 1%?

I am trying gradient boosting on a dataset with event rate about 1% using Enterprise miner, but it is failing to produce any output. My question is, since it a decision tree based approach, is it even ...
1
vote
0answers
30 views

Feature binarization for RF/GBMs?

Are there any advantages to feature binarization for random forests or gradient-boosted machines? For example, suppose I am predicting snowstorms for the next day using various past measurements - ...
1
vote
0answers
22 views

Boosting in unsupervised learning - methods and use cases

I'm looking for methods and uses cases for applying boosting or other ensemble methods for unsupervised learning Examples of such methods and use cases are: Boosting density estimation Saharon ...
3
votes
1answer
35 views

Adaboost: model depth lager than number of predictors

I noticed that I can train models in the R package gbm that have interaction depth larger than the total number of predictors used to train the model. How is it possible that I can train a model of ...
0
votes
0answers
49 views

How to boost the performance of support vector machine?

I have 4 different data samples: Stage 1: [152 X 27578] Stage 2: [48 X 27578] Stage 3: [48 X 27578] Cancer: [63 X 27578] Each sample are the different stage of cancer in descending order. Here I ...
1
vote
0answers
123 views

xgboost binary logistic regression

I am having problems running logistic regression with xgboost that can be summarized on the following example. Lets assume I have a very simple dataframe with two predictors and one target variable: ...
0
votes
0answers
26 views

Boosting the prediction results in machine learning

I am having the datasets of 152 samples and 151 features.I implemented libSVM algorithm as a classifier. I am getting a classification accuracy just above 55% Is there any way I can boost my ...
2
votes
0answers
24 views

Can a decision tree automatically detect the effect on the dependent variable from the product/quotient of two independent variables?

For example, when I use the xgboost algorithm, there are two continuous variables X1 and X2, do I need to specify the product X1*X2 explicitly at the beginning? Or the algorithm can automatically pick ...
2
votes
1answer
37 views

Boosted Trees: Objective Function clarification

Reading through this overview of boosted trees, I'm having trouble understanding how the second line was derived. $$ Obj(t)=\sum_1^n{loss(y_{i} - \hat{y}_i^{(t)})} + \sum_1^t{\Omega(f_i)} \\ = ...
0
votes
0answers
11 views

Boosting with weights estimated by (regularized) linear regression

In the gradient boosting algorithm, the sum of weak learners $$F(x) = \sum_{i=1}^M \gamma_i h_i(x)$$ is, according to Hastie et. al., found via the greedy Forward Stagewise Additive Modeling ...
2
votes
1answer
637 views

XG Boost vs Random Forest for Time Series Regression Forecasting

I am using R's implementation of XGboost and Random forest to generate 1-day ahead forecasts for revenue. I have about 200 rows and 50 predictors. (As I go further in time I have more data so more ...
13
votes
4answers
356 views

“Semi supervised learning” - is this overfitting?

I was reading the report of the winning solution of a Kaggle competition (Malware Classification). The report can be found in this forum post. The problem was a classification problem (nine classes, ...
0
votes
1answer
44 views

extract glmboost model coefficient

I have a model fitted with glmboost function from mboost package. The object name of the fitted model is modelResult. When trying to extra the coefficient of the model. I observed different results ...
0
votes
0answers
8 views

Does (stochastic) boosting reduce to bagging when learning rate is infinitesimal?

If learning rate is very small, the weight of training data remain constant. So all trees are trained with same set of data. I think a major difference is: in the final prediction, boosting uses ...
6
votes
1answer
114 views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
0
votes
0answers
97 views

Extracting underlying model output from Caret's train() function

I am using the great {caret} package to run a lot of models, however I would like to analyse the model as one usually does having run that model in its own right, ...
3
votes
2answers
504 views

Do I have a situation of overfitting in xgboost on this data? How can I reduce it?

I apply the xgboost algorithm for classification. I perform cross-validation in the training data set in order to find parameters (eta, step size shrinkage, = 0.01, maximum depth of a tree: 14, 1400 ...
22
votes
1answer
985 views

Is this the state of art regression methodology?

I've been following Kaggle competitions for a long time and I come to realize that many winning strategies involve using at least one of the "big threes": bagging, boosting and stacking. For ...
1
vote
1answer
37 views

Modelling a time-series with lags

I have a data set with 200 predictors and 700 observations. It is a regular time series, so 700 days in my case. I want to experiment with lagged variables, which I will create manually and save as ...
1
vote
1answer
58 views

Accuracy reduced with Adaboost

I tried using AdaBoost for my classification which is for emotion classification. Without boosting, Random Forest algorithm gave me 42.41% of accuracy. But when I applied AdaBoost along with Random ...