Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

0
votes
0answers
6 views

Feature selection in xgboost vs GBM in H2O

I am working on a big data set( more than 100 variables) and 30 million observations. I tried to build 100 models with a grid search using both XGBoost and GBM in H2O (Sparkling Water). I realized ...
1
vote
0answers
11 views

How to interpret chart generated by gbm.perf function?

I'm new to GBM.Can you help me to understand the interpretation of gbm.perf function? I used following code in R best.iter = gbm.perf(train, method="cv") & got ...
0
votes
1answer
37 views

Are the same number of trees required to compare Random Forest against GBM?

My training set has 13,737 observations with 53 predictors. I need to compare the accuracy of Random Forest vs. GBM. For Random Forest, I set ntree = 128 [based on ...
0
votes
0answers
7 views

Are the same number of trees required while comparing Random Forest to GBM?

My training set has 13,737 observations with 53 predictors. I need to compare the accuracy of Random Forest and GBM. For Random Forest, I set ntree = 128 [based ...
0
votes
0answers
11 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
0
votes
0answers
13 views

XGBoost tree dump contains lots of empty trees

After fitting a regression model using XGBoost, I want to inspect the individual trees that were built. In the resulting table, I find a lot of 0-depth trees, i.e. trees with only a leaf node, and ...
1
vote
0answers
8 views

Evaluation of Classifier Performance on Imbalanced Dataset with Lift Chart

I trained a classifier on imbalanced dataset (label={0,1}) by assigning higher weight to rare event(label=1). Lift chart shows that the predicted and actual curves are very separated. I also trained ...
0
votes
0answers
13 views

AdaBoost gets “stuck”, fails to converge [on hold]

I'm attempting to implement Viola-Jones using AdaBoost in Python. During the AdaBoost step, I add each weak learner to my strong learner, check whether the strong learner's FPR is below a certain ...
2
votes
1answer
13 views

binomial responses in h2o gbm

I am modeling the probability of success in a dataset where I have a both the number of trials and the number of successes (and, obviously, I am modeling $p_i=\frac{total successes}{total trials}$). I ...
1
vote
0answers
10 views

Learning Rate impact on model building time

I wanted to know that does learning rate impact the model building time in case of Gradient Boosted Trees. I do understand that increasing the number of trees have an impact( more the trees, more the ...
2
votes
2answers
45 views

Why are all predictions made by XGBoost distinct?

If I understood correctly the XGBoost is a framework that operates on gradient tree boosting. It means that behind the scenes, it uses a decision tree to make a prediction. So, from what I read in the ...
1
vote
1answer
14 views

Understanding `score` in LightGMB

I'm newly introduced to the LightGBM for a regression problem. Having read the documentation of LightGBM (here), I got puzzled about the ...
1
vote
0answers
30 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
0
votes
0answers
21 views

How do Gradient Boosted Trees calculate errors in classification? [duplicate]

I understand how gradient boosting works for regression when we build the next model on the residual error of the previous model - if we use for example linear regression then it will be the residual ...
2
votes
0answers
25 views

Implausible variable importance for GBM survival: constant difference in importance [closed]

I have a question about a GBM survival analysis. I'm trying to quantify variable importances for my variables (n=453), in a data set of 3614 individuals. The resulting graph wi th variable importances ...
0
votes
0answers
20 views

ada model- variables overall importance

I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees. I would like to have a ...
1
vote
1answer
16 views

Whats is the difference between using risk() and cvrisk() in the R package mboost

I am currently running an additive model using the function gamboost() in the package mboost. When using the ...
0
votes
0answers
28 views

Why AdaBoost works exactly the way it does

I understand the basic idea of AdaBoost -- when training weak classifiers, use more of the difficult examples. However, it puzzles me why I sould modify the weights the way AdaBoost does. There are, ...
0
votes
0answers
18 views

Conditions for Adaboost to perform well

Under which conditions does the AdaBoost algorithm yield good results even on weak learners (i.e. slightly better than random classifiers)?
1
vote
1answer
41 views

(Low cardinality) categorical features handling in gradient boosting libraries

In some popular gradient boosting libraries (lgb, catboost), they all seems like can handle categorical inputs by just specifying the column names of the categorical features, and pass it into a ...
1
vote
1answer
22 views

Differences between “in-bag” and “out-of-bag” empirical risks in the R package “mboost”

currently I am using the mboost R-package to estimate some additive models. When using the function gamboost(), you can control the hyper-parameters for boosting by using the option boost_control(). ...
0
votes
0answers
33 views

how to extract rules from final model made by caret

I have a made cross-validation (k=5) by caret package using C5.0 method. I have 21 features and 7000 instances. The C5.0 trials default is 40. The problem is C5.0 made > 1600 rules over 40 trials, ...
2
votes
1answer
28 views

How do I perform leave one out cross validation with boosting?

I'm working with the Anderson Iris data set and it is too small To split into a test and training set.I use boosting To make a classifier For determining the species Of flower Based on Variables in ...
1
vote
1answer
50 views

Why does GBM package make different predictions for the same data point (after factor issue is fixed)?

I tried to use a GBM model to make predictions for the same data point, but it gave me very different answers. Please see the example below. When using the entire dataset for predicting the first data ...
1
vote
1answer
38 views

How well gradient boosting can predict outside training values domain?

It has been said(link , link) that gradient boosting can predict values that fall outside of training domain for $Y$ in a regression problem. I intuitively sense that there is a distinction between ...
2
votes
1answer
45 views

Xgboost and repeated measures

I am learning xgboost and am planning on running a tree model. My dataset includes repeated measures. In a GLMM I would include the ID to account for repeated measures and I'm curious if I should do ...
2
votes
0answers
20 views

Applying boosting to predictions from a Random Forest

I have a class of datasets for a binary classification problem where it is known that Random Forest performs poorly compared with GBM or FFNN, rarely adding anything to an ensemble. I've had an idea ...
1
vote
0answers
17 views

Boosting using strong learners [duplicate]

At a high level, boosting is the process of adding many weak learners to form a strong learner. My professor had mentioned that in the case of boosting using regression trees, trees with a depth of ...
0
votes
0answers
19 views

Finding Null distribution for gbm interactions

I am trying to determine which interactions in a gbm model are significant using the method described in Friedman and Popescu 2008. My gbm is a classification model with 9 different classes. I'm ...
1
vote
1answer
24 views

How should I formulate the loss function/objective for this predictive modeling problem?

Let's say I have a big department store, selling all kinds of products, like clothing, shoes, cosmetics and electronics, etc. The data I have are daily sales by each item, like ...
0
votes
0answers
18 views

Predict malfunctionings with classifiers trained on truncated data?

I have an historical data-set of machines malfunctionings. I have data from different sensors, and a response variable of malfunctioning or not (1/0). I have difficulties in creating a classifier ...
0
votes
0answers
30 views

Risk classification with non uniform risk exposure (truncated observation period)?

I have N individuals who are exposed to a risk. The risk exposure is in year fractions (from 1/365 to 1). More or less 1/4 of the observations are exposed to the risk for an entire year. I want to ...
0
votes
0answers
18 views

Partial dependence plot R yields lower estimates

I'm computing a classification gbm using R ...
0
votes
0answers
18 views

Matching Pursuit & Boosting: Exponential Convergence?

In the paper by Bühlmann BOOSTING FOR HIGH-DIMENSIONAL LINEAR MODELS, he introduces an algorithm, called componentwise linear least squares, and relates it to the Matching Pursuit algorithm by Mallat &...
2
votes
0answers
23 views

In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
3
votes
1answer
104 views

Please correct my assumption on how regression trees work

I'm trying to understand how regression trees work, I've been experimenting with catboost and xgboost in python, and I'm getting results which I don't expect, can someone please clarify (and apologies ...
0
votes
1answer
51 views

Is the best model always one with best test score, even though it looks overfit?

I'm making a binary classification model using gradient boosting (lightgbm). I usually use learning curves to check if my model is overfitting. The metric I'm using is sklearn's average precision-...
0
votes
0answers
20 views

GBDT- randomized repetition feature selection

Consider the following approach for feature selection in the specific case of gradient boosting decision trees: Randomly pick X% of features Run algorithm Record importance of each feature Repeat ...
0
votes
0answers
27 views

How to prevent GBDT from splitting on uninformative features?

I'm looking into using feature importance scores from GBDT for feature selection. Although GBDT does not need manual feature selection, the number of features is a restriction of the production system ...
2
votes
0answers
221 views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
4
votes
1answer
88 views

Did I understand AdaBoost correctly?

My mantra has always been that if you are not able to recreate something you haven't really understood it. In this manner I tried to implement the AdaBoost algorithm of Freund and Schapire I used one ...
4
votes
2answers
119 views

Machine Learning - Prediction Interval - Cheating?

I work at a company that is trying to use machine learning methods in particular gradient boosting and neural networks to make predictions on stock market data, so using historical data to predict ...
1
vote
1answer
156 views

Weak learners for XGBoost with Tweedie distribution

Could you please explain what are the standard weak learners for XGBoost when the objective parameter equals reg:tweedie? Are they GLMs (with Tweedie distribution of dependent variable) on all ...
0
votes
1answer
119 views

H2O GBM and Caret GBM

Hi I have doubt regarding the interaction. depth parameter in caret. I found a useful link hereabout interaction.depth in caret Now I am trying to find the similar parameter in H2O-GBM . Can anyone ...
0
votes
1answer
15 views

Using RF/GBM regressors when some observables are not real valued but just greater than a given value?

Say I can experimentally measure some number of $N$ data points which have an observable $y$ value I'm trying to model where $y=[0, 100)$ and is a continuously valued number. However, I also have $M$ ...
1
vote
1answer
147 views

Gradient boosting regression trained on skewed data

My target feature is right-skewed. I want to apply gradient boosting regression algorithm to predict it but I'm not sure what kind of preprocessing should I apply. As gradient boosting is based on ...
0
votes
0answers
16 views

Optimizing recursive loss functions with decision trees

For time series applications, it is often helpful to model things in a recursive fashion. For instance, let $f(x)$ be a model which predicts the next time step of some time series, so that $$ f(x^{n+...
1
vote
1answer
54 views

Does it make sense to use PCA right after GBM?

My Problem: I'm trying to classify a data into two groups as A and B based on 25 observations (data point) and 100 features. I used the Gradient Boosting Machine (GBM) to find out which feature has ...
2
votes
1answer
145 views

Gradient Boosted Trees vs Neural Network for limited data [closed]

I have a classification problem, with about 10 different inputs, some boolean, some categorical (and unrelated to each other), some being a float between 0 and 1, which need to be mapped to 4 ...
1
vote
0answers
26 views

Boosted decision trees: in which situations are “deep” decision trees performing better?

The general idea of boosted decision trees is to use very simple trees in the following manner (simplified, for intuition only): start with a simple tree, fit another simple tree on the residuals, ...