Gradient Boosting Machine

learn more… | top users | synonyms

0
votes
0answers
6 views

How does gbm (in r) control the sub-sampling of predictor variables? [on hold]

In gbm, the parameter, bag fraction, is used to randomly select subset of observations. And the sampling could be contorlled by set.seed() before gbm (Please correct me if I was wrong). But is there ...
0
votes
0answers
7 views

Why is xgboost giving bad scores and taking a very long time? [migrated]

I am completely new to xgboost so thank you very much for any help in advance. I have this simple script. ...
0
votes
2answers
24 views

Getting information from a GBM model

I built a gbm model (using caret package) in order to predict the probability of someone buy or not a car. However as this kind of model act as a black box my issue is to replicate the "profile" of ...
1
vote
0answers
16 views

Getting prediction intervals from GBM models

I am working with GBM regression models(in H2O) and am using Quantile distribution for the distribution parameter. I am looking for a method to provide prediction intervals in addition to point value ...
0
votes
1answer
15 views

Generalised Boosted Models (GBM) Assumptions

I have a rather simple question the answer to which I struggle to find in any literature about GBM. I am fitting a GBM model as per G.Ridgeway (2007), paper can be found in http://www.saedsayad.com/...
3
votes
0answers
68 views

Splitting data for train/test for time series

A week ago or so I was at a conference. Long story short, I ran into a friend who is quite good at machine learning so I asked them a question about why I might be getting what I think is poor fit on ...
0
votes
0answers
21 views

Plot GBM interpretation

I don't really understand what does plot.gbm produces. The description on CRAN is : " Plots the marginal effect of the selected variables by integrating out the other variables ". For example, on this ...
0
votes
0answers
31 views

About GBM function parameters (Bag.fraction, interaction.depth)

I don't understand how the gbm's parameter "bag.fraction" works. For me, gradient boosting works globally like that : Fit a tree f_hat_b with d splits to the training data (X, r) (where r_i=y_i for ...
0
votes
1answer
27 views

xgboost: does it support stochastic gradient boosting? [closed]

What function/parameter needs to be set to enable stochastic gradient boosting in XGBoost? I ask because in https://statweb.stanford.edu/~jhf/ftp/stobst.pdf, Jerome Friedman shows that stochastic ...
0
votes
0answers
11 views

Interpretation of `int.size` values from `gbm.interactions` in R

Does anyone know how to interpret the int.size values from the gbm.interactions output? Do they say anything about significance of the interaction? In the table below, can any of these interactions ...
0
votes
1answer
45 views

Improving a boosted regression model or change?

I am looking at a data set that contains multiple predictors and a continuous response. Using dismo along with gbm I built (a terrible one?) model. Using the package sROC, I got an AUC or 0.48 - so my ...
2
votes
1answer
55 views

GBM: impact of the loss function

I'm familiar with Random Forestand Adaboost and if I understand well the advantage of GBM on ...
0
votes
0answers
14 views

gradient boosted cart models - paradigm

Disclaimers: This is a question about the model, not the software. While I was exposed to boosting in my undergraduate program (2007) I have been using boosted trees on and off for the last ~4 years. ...
8
votes
4answers
586 views

Gradient boosting machine accuracy decreases as number of iterations increases

I'm experimenting with the gradient boosting machine algorithm via the caret package in R. Using a small college admissions dataset, I ran the following code: <...
0
votes
0answers
29 views

Multinomial loss derivative in gradient boost

I'm struggling hard to understand the derivative Jerome Friedman uses to extend gradient boosting to the multiclass case using multinomial logistic loss in his paper on Gradient Boosting: Greedy ...
3
votes
3answers
126 views

Prediction intervals for machine learning algorithms

I want to know if the process described below is valid/acceptable and any justification available. The idea: Supervised learning algorithms don't assume underlying structures/distributions about the ...
2
votes
0answers
234 views

choosing a loss function for gbm

I am using gbm to predict an imbalanced binary outcome, with the intent of obtaining a ranking by class probability estimation that produces a strong class ...
0
votes
0answers
14 views

Stacking GBT using Logistic Regression

1) I build Gradient Boosted Tree Model in h2o and now i have the POJO. 2) I extracted the weight for each tree of GBT for my population 3) I used the extracted weight to train a logistic regression ...
0
votes
0answers
16 views

Correct Feature Selection Methodology?

I am running a weighted multiple linear regression where my independent variables take binary values, 0 and 1. The dependent variable y, takes numeric values (positive as well as negative). The ...
3
votes
1answer
124 views

Difference in partial dependence calculated by R and Python

I noticed there's a difference in partial dependence calculated by R package gbm and Python's scikit-learn. Here's ...
3
votes
0answers
46 views

For a continuous DV that ranges between 0 and 1, which of the gbm package's distributions would you recommend?

My DV consists of a variable ranging between 0 and 1 (it's a percentage to be more precise) and I am using the gbm package to predict it. Given the nature of my DV ...
1
vote
1answer
107 views

Feature subsampling with gradient boosting

A key component in building random forest models is feature subsampling, i.e., building each individual tree with only a percentage of predictors chosen randomly by tree. The literature often ...
1
vote
1answer
32 views

How to upweight an arbitray feature in GBM?

Are there any ways to upweight any particular feature for GBM (tree boosting)? The motivation is: Assuming one would like to put a few dummy indicators, e.g., X_A, X_B, X_C, ..., into GBM, for the ...
0
votes
0answers
33 views

Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
0
votes
1answer
38 views

Optimizing cumulative lift in classification

Suppose I have a business problem where I can reach out to 10% of my customers to prevent them from churning. I want to capture as much of the high risk customers I can. Let's say I'm tuning a random ...
1
vote
0answers
88 views

How to tune the weak learner in boosted algorithms

It is commonly said that boosted algorithms (adaboost, gradient boosted trees) are composed of many "weak" learners. Let's stick to decision trees as the base learners. Some empirical studies ...
2
votes
2answers
34 views

Ordinal vs Regular Factors in R - is there a difference in modeling

Can someone explain to me or show me an example of the difference between an ordinal variable and a regular categorical variable in R and how the outcome differs? Is there even a difference at all? ...
0
votes
0answers
61 views

Is my labeled data bad if I get good cross validated results but my model performs poorly on new, unlabeled data?

I'm learning a gradient boosting model with decision trees using h2o. I've spent a lot of time prepping some data for classification. It's a big data problem and I actually have around 20 million ...
1
vote
0answers
48 views

dropout regularization in gbm

Background: Dropout regularization reduces overfitting in Neural networks, especially deep belief networks (srivastava14a). It also has the opportunity to accelerate learning because individual ...
2
votes
1answer
108 views

GBM and highly correlated predictors

I have a data set with 70 predictors, 68 numeric and 2 factors. When I build a gbm model using all predictors, I get an R2 of 0.767 and RMSE of 175.15. I get similar numbers on the training and the ...
0
votes
0answers
21 views

load GBM and show summary

i run GBM codes, and get relative.influence by "summary(GBMmodel,n.trees=best.iter,plotit=FALSE)" Then, I save the file to "123.R", close it. When I re-open R, load the file, however, how to obtain ...
1
vote
1answer
61 views

What parameter of GBM does gradient descent update after calculating gradient of loss function?

I am going through Elements of statistical Learning and trying to understand GBM algorithm. The algorithm of GBM is shown below. I understand general gradient descent algorithm mentioned below very ...
0
votes
2answers
85 views

how to plot 3D partial dependence in GBM

I can use the following code to get one-dimensional partial dependence plot. what code can I plot two-variable partial dependence plot, that's the three dimensional figure. Thanks. plot.gbm(GBMmodel,...
0
votes
1answer
99 views

marginal plots in GBM

plot.gbm (...) Description: plots the marginal effect of the selected variables by "intergrating" out the other variables. could you please put it another way? the same meaning with "parital ...
0
votes
0answers
29 views

which interaction depth should be specified in GBM

GBMmodel = gbm(mydataset~x1+x2+x3+x4+x5, data=mydataset,distribution="gaussian",n.trees=1500,shrinkage=0.005,interaction.depth=3, bag.fraction=0.75,train.fraction=0.75,n.minobsinnode=5,cv.folds=3,keep....
3
votes
1answer
47 views

Adaboost: model depth lager than number of predictors

I noticed that I can train models in the R package gbm that have interaction depth larger than the total number of predictors used to train the model. How is it possible that I can train a model of ...
0
votes
0answers
92 views

Interpretation of partial dependence plots for multinomial GBM

I've been a big fan of the gbm package for some time, but am having difficulty understanding the output from the partial dependence plots in the case for multinomial classification problems. Below ...
1
vote
0answers
40 views

grouping attributes in RF and GBM

i have a dataset with 1000 samples and ~11k features (SNP markers). i have identified 100 additional binary features describing the markers themselves so i have a ...
2
votes
1answer
69 views

What happens to multi-category variables in algorithms like Random Forest that sample the feature space?

Suppose I have a multi-level categorical variable like color (say, with 7 levels). Some software libraries only allow numeric matrices to train models, so we need ...
0
votes
0answers
80 views

Metrics to evaluate the difference between gbm implementation

I am only interested in the case of a binary model. I count three different implementation of the gradient boost model: gbm from the ...
0
votes
0answers
21 views

Predict count data for unsurveyed areas

I am looking to predict count data from deer surveys for the unsurveyed areas. I want to make these predictions based on vegetation type and size of the vegetation type (acres). I started by using <...
3
votes
1answer
130 views

Interpretation of a gradient boost model

I recently did a gradient boost model to predict an event Y/N. I have a lot of features and a huge dataset. After a grid search cross validation, I manage to get an efficient enough model. (It is the ...
0
votes
2answers
160 views

Parallelizing GBM in R

I'm able to parallelize randomForest based on the foreach function in the following way: ...
1
vote
1answer
129 views

rebuild the tree generated by GBM package in R manually

I've used the gbm in R to generate a model. Although I can use predict.gbm to fit the model on new data set, I want to know the detailed step of gbm to calculate the prediction, beacuse I need to ...
0
votes
1answer
121 views

How is gbm package different from caret with gbm method?

I have a gbm problem and I am using the gbm package in R for it. But in most forums I see people using caret package for gbm. Is there any advantage of using caret instead of gbm package? If so, what ...
1
vote
1answer
47 views

gbm.fit() in R language

I have a data with size of 1200 rows having binary dependent variable and around 20 independent variables which are categorical as well as continuous in nature. I have tried machine learning technique ...
1
vote
1answer
47 views

Equal sampling for machine learning

I have a data with size of 1200 rows having binary dependant variable and around 20 independant variables which are categorical as well as continous in nature. I have tried 2 machine learning ...
1
vote
1answer
49 views

How to handle non- linear predictors in Decision Trees

I have a data set of just 1800 rows. I am not sure about how to provide inputs for certain variables. For eg., I have a column called number of previous jobs, and the values lie between 1 and 10, with ...
1
vote
2answers
264 views

Why the error on a training set is decreasing, while the error on the validation set is increasing?

When training XGboost model I observe the following outputs: ...
0
votes
1answer
44 views

Does the prediction/Scoring of model depends on the sequence of the variable?

Let's assume my dataset D1 has the variables x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 I have used Gradient boosting regression on this and want to score on a new ...