Boosting is a process of finding & combining weakly predictive models into a strongly predictive model.

learn more… | top users | synonyms

0
votes
0answers
42 views

How to boost the performance of support vector machine?

I have 4 different data samples: Stage 1: [152 X 27578] Stage 2: [48 X 27578] Stage 3: [48 X 27578] Cancer: [63 X 27578] Each sample are the different stage of cancer in descending order. Here I ...
1
vote
0answers
62 views

xgboost binary logistic regression

I am having problems running logistic regression with xgboost that can be summarized on the following example. Lets assume I have a very simple dataframe with two predictors and one target variable: ...
0
votes
0answers
23 views

Boosting the prediction results in machine learning

I am having the datasets of 152 samples and 151 features.I implemented libSVM algorithm as a classifier. I am getting a classification accuracy just above 55% Is there any way I can boost my ...
2
votes
0answers
17 views

Can a decision tree automatically detect the effect on the dependent variable from the product/quotient of two independent variables?

For example, when I use the xgboost algorithm, there are two continuous variables X1 and X2, do I need to specify the product X1*X2 explicitly at the beginning? Or the algorithm can automatically pick ...
2
votes
1answer
29 views

Boosted Trees: Objective Function clarification

Reading through this overview of boosted trees, I'm having trouble understanding how the second line was derived. $$ Obj(t)=\sum_1^n{loss(y_{i} - \hat{y}_i^{(t)})} + \sum_1^t{\Omega(f_i)} \\ = ...
0
votes
0answers
8 views

Boosting with weights estimated by (regularized) linear regression

In the gradient boosting algorithm, the sum of weak learners $$F(x) = \sum_{i=1}^M \gamma_i h_i(x)$$ is, according to Hastie et. al., found via the greedy Forward Stagewise Additive Modeling ...
0
votes
0answers
155 views

XG Boost vs Random Forest for Time Series Regression Forecasting

I am using R's implementation of XGboost and Random forest to generate 1-day ahead forecasts for revenue. I have about 200 rows and 50 predictors. (As I go further in time I have more data so more ...
13
votes
4answers
276 views

“Semi supervised learning” - is this overfitting?

I was reading the report of the winning solution of a Kaggle competition (Malware Classification). The report can be found in this forum post. The problem was a classification problem (nine classes, ...
0
votes
1answer
30 views

extract glmboost model coefficient

I have a model fitted with glmboost function from mboost package. The object name of the fitted model is modelResult. When trying to extra the coefficient of the model. I observed different results ...
0
votes
0answers
7 views

Does (stochastic) boosting reduce to bagging when learning rate is infinitesimal?

If learning rate is very small, the weight of training data remain constant. So all trees are trained with same set of data. I think a major difference is: in the final prediction, boosting uses ...
6
votes
1answer
66 views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
0
votes
0answers
39 views

Extracting underlying model output from Caret's train() function

I am using the great {caret} package to run a lot of models, however I would like to analyse the model as one usually does having run that model in its own right, ...
3
votes
2answers
198 views

Do I have a situation of overfitting in xgboost on this data? How can I reduce it?

I apply the xgboost algorithm for classification. I perform cross-validation in the training data set in order to find parameters (eta, step size shrinkage, = 0.01, maximum depth of a tree: 14, 1400 ...
20
votes
1answer
904 views

Is this the state of art regression methodology?

I've been following Kaggle competitions for a long time and I come to realize that many winning strategies involve using at least one of the "big threes": bagging, boosting and stacking. For ...
1
vote
1answer
28 views

Modelling a time-series with lags

I have a data set with 200 predictors and 700 observations. It is a regular time series, so 700 days in my case. I want to experiment with lagged variables, which I will create manually and save as ...
1
vote
1answer
41 views

Accuracy reduced with Adaboost

I tried using AdaBoost for my classification which is for emotion classification. Without boosting, Random Forest algorithm gave me 42.41% of accuracy. But when I applied AdaBoost along with Random ...
5
votes
1answer
53 views

Boosting neural networks

Well recently I was working on learning boosting algorithms, such as adaboost, gradient boost, and I have known the fact that the most common used weak-learner is trees. I really want to know are ...
0
votes
0answers
6 views

Drawbacks of replacing a factor by average value of the target

I have to encode a lot of factors before feeding them to a gradient boosting algorithm. I noticed that the best method was to encode them replacing them with the average value of the target when this ...
1
vote
2answers
168 views

Why the error on a training set is decreasing, while the error on the validation set is increasing?

When training XGboost model I observe the following outputs: ...
0
votes
0answers
313 views

Conducting pairwise ranking with XGBoost

I am trying to build a ranking model using xgboost, which seems to work, but am not sure however of how to interpret the predictions. I haven't been able to find relevant documentation or examples on ...
1
vote
0answers
267 views

How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
0
votes
0answers
59 views

When do we use gblinear versus gbtree?

When do we use gblinear boosting vs gbtree boosting in xgboost library. I have a metereological rain data with lots of missing values.
1
vote
1answer
25 views

How can I improve it with regularization?

Why too large value of lambda causes my data to underfit in gradient descent boosting trees? How can I improve it with regularization?
0
votes
1answer
36 views

Does the prediction/Scoring of model depends on the sequence of the variable?

Let's assume my dataset D1 has the variables x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 I have used Gradient boosting regression on this and want to score on a new ...
0
votes
1answer
104 views

how to interpret gradient boost model & results in R?

I'm using R to model votes. I've found that the extreme gradient boost xgbTree algorithm gives nice results. I'm a newbie and don't really know how to interpret the ...
0
votes
0answers
29 views

Interpreting Gradient Boosting Machine: Feature importance undervalues binary features, overvalues continuous features

I'm migrating this question over from stackoverflow because it has more to do with stats/ml than scikit learn itself. I'm using sci-kit learn's gradient boosting machine for a binary classification ...
0
votes
0answers
27 views

Gradient boosting for multitask learning

Let's say I have relation learning task similar to what is solved here: http://arxiv.org/abs/1403.6652 Simply put, I have a social graph with some nodes marked up with some tags, and I have to infer ...
0
votes
0answers
31 views

boosted regression tree method for panel data?

In my current research I have a panel dataset that contains observations of one dependent (vegetation growth: trend component over time) and multiple independent variables obtained over multiple time ...
0
votes
1answer
57 views

LogitBoost R in package “caTools”, which algo is it?

What is the name, and where can i find documentation on the algorithm used in LogitBoost for the "one node decision trees"? R, package = caTools
4
votes
1answer
84 views

Definition of complexity of a tree in xgboost

Doing research about the xgboost algorithm I went through the documentation. In this approach trees are regularized using the complexity definition $$ \Omega(f) = \gamma T + \frac12 \lambda ...
2
votes
1answer
95 views

Reference for xgboost

The xgboost package and the algorithm behind it are often mentioned in data science competitions. The method is called extreme gradient boosting. How does this differ from usual gradient boosting? I ...
3
votes
1answer
128 views

Best practicies for coding Categorical features for Decision Trees?

When coding categorical features for linear regression, there is a rule: number of dummies should be one less than the total number of levels (to avoid collinearity). Does there exist a similar rule ...
1
vote
0answers
30 views

Tolerance in boosted regression trees

I was interested in knowing if anyone is using the custom made function of BRT by Elith et al. (2008) in Journal of Animal Ecology "A working guide to boosted regression trees" and knows what does ...
1
vote
1answer
33 views

Diagnosing linear skew of residuals in boosted tree

I have trained a boosted tree regression with the following code (out of the caret and gbm packages: ...
1
vote
1answer
52 views

Which classifiers or anomaly detectors work well with little data?

I'm trying to predict whether a drawing matches a "trained" drawing. The drawings are black on white background images (32 x 32). More specifically I'd like to check if a new example is relatively ...
0
votes
0answers
28 views

Evaluating which decision model/approach performs best

I'm trying to build decision models using an imbalanced dataset that's n = 300..... I know - I don't like it either :-( Thanks to advice from a few reditters, I tried out multiple options including ...
5
votes
4answers
202 views

Which data are used at each step of Stochastic Gradient Boosting? Subsample of the original training set or gradient of the loss function?

The bag.fraction parameter in SGB controls the size of the random subsample of the original training set on which each successive weak learner is fitted: At ...
1
vote
0answers
14 views

Model averaging for negative gradient boosting?

I'm using negative gradient boosting for Cox regression. After finding out the optimum lambda value by evaluating the empirical risk of 10-fold cross validation, I want to determine the final model ...
2
votes
1answer
383 views

Gradient boosting decision tree implementation

I am willing to implement my own GBM. I have been looking - unsuccessfully - for a clear article describing the implementation of gradient boosting machine for decision trees. Sources like this are ...
10
votes
2answers
4k views

How to tune hyperparameters of xgboost trees?

I have a class imbalanced data & I want to tune the hyperparameters of the boosted tress using xgboost. Questions Is there an equivalent of gridsearchcv or randomsearchcv for xgboost? If not ...
0
votes
1answer
44 views

Methods for supervised regression learning

What ML methods should I try for a data set of around 1000 samples? The output variable is dependent on say 10 regressors of which 8 are real numbers and the other 2 are categorical. Or they might ...
2
votes
0answers
106 views

Boosting: why is the learning rate called a regularization parameter?

The learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically a shallow tree- that is added in the series. It was shown to dramatically ...
0
votes
0answers
59 views

error family in boosted regression tree: gbm package

I am trying to understand boosted regression tree. I am using the gbm package in R. I noticed that in this package one has to specify the error family. I understand ...
1
vote
0answers
14 views

What is the best achieved/possible score for KDD 2009 with Boosted Decision Trees? [closed]

Could you please let me know what is the best achieved or possible score for KDD 2009 churn dataset with Boosted Decision Trees?
0
votes
0answers
44 views

derive loss function for gamma regression

In the R package mboost there is a family called "GammaReg" which implementes "negative Gamma log-likelihood with logarithmic link function". Still, I don't really ...
4
votes
1answer
461 views

Intuitive explanations of differences between Gradient Boosting Trees (GBM) & Adaboost

I'm trying to understand the differences between GBM & Adaboost. These are what I've understood so far: There are both boosting algorithms, which learns from previous model's errors and ...
2
votes
1answer
189 views

How many AdaBoost iterations?

In one R package, ada, the main AdaBoost fitting function (also called ada) takes an argument specifying the number of boosting ...
1
vote
1answer
172 views

The relationship between adaboost and gradient boosting

I am reading the chapter 10 of "The Elements of Statistical Learning 2nd ed, (ESLII)", where the Adaboost algorithm is explained by minimizing the exponential loss using stagewise additive modelling ...
2
votes
0answers
168 views

Modelling clustered data using boosted regression trees

I'm modelling habitat selection using boosted regression trees (BRTs), which I prefer over linear models for a variety of reasons (modeling complex nonlinear relationships and interactions, ...
2
votes
1answer
44 views

Notation - gradients

I am reading Friedman's paper on stochastic gradient boosting - and saw as per below the following set of formulas. Purely from a notation perspective - why do we write $F(x)$ = $F_{m-1}(x)$ outside ...