Gradient Boosting Machine

learn more… | top users | synonyms

0
votes
0answers
18 views

Multinomial loss derivative in gradient boost

I'm struggling hard to understand the derivative Jerome Friedman uses to extend gradient boosting to the multiclass case using multinomial logistic loss in his paper on Gradient Boosting: Greedy ...
3
votes
3answers
109 views

Prediction intervals for machine learning algorithms

I want to know if the process described below is valid/acceptable and any justification available. The idea: Supervised learning algorithms don't assume underlying structures/distributions about the ...
2
votes
0answers
72 views

choosing a loss function for gbm

I am using gbm to predict an imbalanced binary outcome, with the intent of obtaining a ranking by class probability estimation that produces a strong class ...
1
vote
0answers
14 views

R: Plot trees from h2o.randomForest() and h2o.gbm() [migrated]

Looking for an efficient way to plot trees in rstudio, H2O's Flow or in local html page from h2o's RF and GBM models similar to the one in the image in link below. Specifically, how do you plot trees ...
0
votes
0answers
7 views

Stacking GBT using Logistic Regression

1) I build Gradient Boosted Tree Model in h2o and now i have the POJO. 2) I extracted the weight for each tree of GBT for my population 3) I used the extracted weight to train a logistic regression ...
0
votes
0answers
13 views

Correct Feature Selection Methodology?

I am running a weighted multiple linear regression where my independent variables take binary values, 0 and 1. The dependent variable y, takes numeric values (positive as well as negative). The ...
3
votes
0answers
60 views

Difference in partial dependence calculated by R and Python

I noticed there's a difference in partial dependence calculated by R package gbm and Python's scikit-learn. Here's ...
3
votes
0answers
40 views

For a continuous DV that ranges between 0 and 1, which of the gbm package's distributions would you recommend?

My DV consists of a variable ranging between 0 and 1 (it's a percentage to be more precise) and I am using the gbm package to predict it. Given the nature of my DV ...
1
vote
1answer
74 views

Feature subsampling with gradient boosting

A key component in building random forest models is feature subsampling, i.e., building each individual tree with only a percentage of predictors chosen randomly by tree. The literature often ...
1
vote
1answer
25 views

How to upweight an arbitray feature in GBM?

Are there any ways to upweight any particular feature for GBM (tree boosting)? The motivation is: Assuming one would like to put a few dummy indicators, e.g., X_A, X_B, X_C, ..., into GBM, for the ...
0
votes
0answers
27 views

Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
0
votes
1answer
29 views

Optimizing cumulative lift in classification

Suppose I have a business problem where I can reach out to 10% of my customers to prevent them from churning. I want to capture as much of the high risk customers I can. Let's say I'm tuning a random ...
1
vote
0answers
66 views

How to tune the weak learner in boosted algorithms

It is commonly said that boosted algorithms (adaboost, gradient boosted trees) are composed of many "weak" learners. Let's stick to decision trees as the base learners. Some empirical studies ...
2
votes
2answers
32 views

Ordinal vs Regular Factors in R - is there a difference in modeling

Can someone explain to me or show me an example of the difference between an ordinal variable and a regular categorical variable in R and how the outcome differs? Is there even a difference at all? ...
0
votes
0answers
46 views

Is my labeled data bad if I get good cross validated results but my model performs poorly on new, unlabeled data?

I'm learning a gradient boosting model with decision trees using h2o. I've spent a lot of time prepping some data for classification. It's a big data problem and I actually have around 20 million ...
1
vote
0answers
28 views

dropout regularization in gbm

Background: Dropout regularization reduces overfitting in Neural networks, especially deep belief networks (srivastava14a). It also has the opportunity to accelerate learning because individual ...
2
votes
1answer
77 views

GBM and highly correlated predictors

I have a data set with 70 predictors, 68 numeric and 2 factors. When I build a gbm model using all predictors, I get an R2 of 0.767 and RMSE of 175.15. I get similar numbers on the training and the ...
0
votes
0answers
17 views

load GBM and show summary

i run GBM codes, and get relative.influence by "summary(GBMmodel,n.trees=best.iter,plotit=FALSE)" Then, I save the file to "123.R", close it. When I re-open R, load the file, however, how to obtain ...
1
vote
1answer
59 views

What parameter of GBM does gradient descent update after calculating gradient of loss function?

I am going through Elements of statistical Learning and trying to understand GBM algorithm. The algorithm of GBM is shown below. I understand general gradient descent algorithm mentioned below very ...
0
votes
1answer
58 views

how to plot 3D partial dependence in GBM

I can use the following code to get one-dimensional partial dependence plot. what code can I plot two-variable partial dependence plot, that's the three dimensional figure. Thanks. ...
0
votes
1answer
61 views

marginal plots in GBM

plot.gbm (...) Description: plots the marginal effect of the selected variables by "intergrating" out the other variables. could you please put it another way? the same meaning with "parital ...
0
votes
0answers
23 views

which interaction depth should be specified in GBM

GBMmodel = gbm(mydataset~x1+x2+x3+x4+x5, data=mydataset,distribution="gaussian",n.trees=1500,shrinkage=0.005,interaction.depth=3, ...
3
votes
1answer
42 views

Adaboost: model depth lager than number of predictors

I noticed that I can train models in the R package gbm that have interaction depth larger than the total number of predictors used to train the model. How is it possible that I can train a model of ...
0
votes
0answers
67 views

Interpretation of partial dependence plots for multinomial GBM

I've been a big fan of the gbm package for some time, but am having difficulty understanding the output from the partial dependence plots in the case for multinomial classification problems. Below ...
1
vote
0answers
39 views

grouping attributes in RF and GBM

i have a dataset with 1000 samples and ~11k features (SNP markers). i have identified 100 additional binary features describing the markers themselves so i have a ...
2
votes
1answer
63 views

What happens to multi-category variables in algorithms like Random Forest that sample the feature space?

Suppose I have a multi-level categorical variable like color (say, with 7 levels). Some software libraries only allow numeric matrices to train models, so we need ...
0
votes
0answers
63 views

Metrics to evaluate the difference between gbm implementation

I am only interested in the case of a binary model. I count three different implementation of the gradient boost model: gbm from the ...
0
votes
0answers
19 views

Predict count data for unsurveyed areas

I am looking to predict count data from deer surveys for the unsurveyed areas. I want to make these predictions based on vegetation type and size of the vegetation type (acres). I started by using ...
3
votes
1answer
103 views

Interpretation of a gradient boost model

I recently did a gradient boost model to predict an event Y/N. I have a lot of features and a huge dataset. After a grid search cross validation, I manage to get an efficient enough model. (It is the ...
0
votes
1answer
112 views

Parallelizing GBM in R

I'm able to parallelize randomForest based on the foreach function in the following way: ...
1
vote
1answer
93 views

rebuild the tree generated by GBM package in R manually

I've used the gbm in R to generate a model. Although I can use predict.gbm to fit the model on new data set, I want to know the detailed step of gbm to calculate the prediction, beacuse I need to ...
0
votes
1answer
99 views

How is gbm package different from caret with gbm method?

I have a gbm problem and I am using the gbm package in R for it. But in most forums I see people using caret package for gbm. Is there any advantage of using caret instead of gbm package? If so, what ...
1
vote
1answer
43 views

gbm.fit() in R language

I have a data with size of 1200 rows having binary dependent variable and around 20 independent variables which are categorical as well as continuous in nature. I have tried machine learning technique ...
1
vote
1answer
45 views

Equal sampling for machine learning

I have a data with size of 1200 rows having binary dependant variable and around 20 independant variables which are categorical as well as continous in nature. I have tried 2 machine learning ...
1
vote
1answer
44 views

How to handle non- linear predictors in Decision Trees

I have a data set of just 1800 rows. I am not sure about how to provide inputs for certain variables. For eg., I have a column called number of previous jobs, and the values lie between 1 and 10, with ...
1
vote
2answers
220 views

Why the error on a training set is decreasing, while the error on the validation set is increasing?

When training XGboost model I observe the following outputs: ...
0
votes
1answer
40 views

Does the prediction/Scoring of model depends on the sequence of the variable?

Let's assume my dataset D1 has the variables x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 I have used Gradient boosting regression on this and want to score on a new ...
0
votes
0answers
48 views

Calculating odds ratios when using gradient boosting approach

I always report odds ratios when using logistic regression for predictions. I wanted know is it meaningful to report odds ratios when modeling with gradient boosting approach? I am using gbm package ...
0
votes
0answers
74 views

Use lagged variable in tree-based model (GBM or RF)

I use GBM (gradient boosting model, R package: GBM) and RF(randomforest, R package: RandomForest) to predict a 0/1 binary variable (current disease status). My attributes (predictors) includes last ...
1
vote
1answer
63 views

R gbm: Lower shrinkage gives worse results?

I've read here and on other sites that when using GBM in R lowering shrinkage gives better results. Yet in my case this is clearly not the case. 0.1 is better than 0.01 with same amount of trees. Even ...
1
vote
0answers
42 views

Tolerance in boosted regression trees

I was interested in knowing if anyone is using the custom made function of BRT by Elith et al. (2008) in Journal of Animal Ecology "A working guide to boosted regression trees" and knows what does ...
0
votes
1answer
63 views

GBM Performance on different sampling techniques

I am working on a healthcare data set for breast cancer patients. This data set is class imbalances and the distribution of positive and negative classes is 80%/20%. In order to deal with the class ...
1
vote
1answer
274 views

Why use dummy variables in GBM using CARET library in R

I have seen a few examples implemting the gbm algorithm on youtube using the titanic dataset. These examples have turned some factor variables into dummy/indicator variables when GBM can handle factor ...
1
vote
1answer
57 views

can random forest project/interpolate based on new values of X?

Sometimes I want a model to predict what would happen when presented with values of predictor variables that it has not seen before. For example, say, I have predictor variables (X) that go from 1 ...
0
votes
1answer
75 views

how does predicted median go above 95% prediction interval when using GBM with quantile loss function

I was checking out how to create prediction intervals with Gradient boosted regression trees using Scikit-learn. If you set the alpha at .95 or .05, you can get the 95% prediction interval around the ...
7
votes
2answers
3k views

Gradient Boosting Tree vs Random Forest

Gradient tree boosting as proposed by Friedman uses decision trees as base learners. I'm wondering if we should make the base decision tree as complex as possible (fully grown) or simpler? Is there ...
1
vote
0answers
51 views

Does it make sense to minimize AUC when using GBM with weights?

I am using gbm(R's caret packages - using train function) on a class imbalanced data set with weights. So, class-1 has a weight of 1 and class-0 has a weight of 10. I am using parameter tuning and ...
1
vote
0answers
108 views

Meaning of Surrogate Split

I tried to figure but couldn't on what happens when missing values are present in some predictor variable and we have to solve the problem of regression using Random Forest. What is the meaning of ...
2
votes
1answer
149 views

Sampling : Gradient Boosting Tree

I have a question regarding the algorithm of Gradient Boosting Tree. I understand Simple tree is built for only a randomly selected sub sample of the full data set (random without replacement). Each ...
0
votes
0answers
91 views

error family in boosted regression tree: gbm package

I am trying to understand boosted regression tree. I am using the gbm package in R. I noticed that in this package one has to specify the error family. I understand ...