Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

1
vote
1answer
24 views

Regression when target has a wide range

I'm working on a regression model where I have to predict time. These times go from a few seconds to up to 30 min and more. I calculated the sMAPE through 1 minute bins of the target, and noticed ...
2
votes
0answers
18 views

In gradient boosting, what is the thing being boosted called?

Quoting from this answer that explains how to do boosting when a 'link' function is involved: "Instead, we can re-express this as a function of $L_i$, (in this case also known as the log odds) $$ \...
0
votes
0answers
25 views

Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]

I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
0
votes
2answers
32 views

why is number of epochs set as external parameter?

I am confused by the very notion of epochs in neural networks (as well as number of trees in gradient boosting). Gradient descent method (as most optimization algorithms) keep going until the loss ...
0
votes
0answers
22 views

How to convince people that developed predictive model based on Gradient Boosting Machine (GBM) has enough accuracy?

First of all, I'm not a data scientist. I'm an engineer that wants to use machine learning to do a binary classification based on a data that is extracted from computational modeling. I have four ...
2
votes
1answer
41 views

The error keeps decreasing with the increase of number of trees

I want to find optimized number of trees in Gradient Boosting. However, the error keeps decreasing with the increase of number of trees, I set the number of trees over 1000, but the error still keeps ...
1
vote
0answers
31 views

How does offset in XGBoost is handled in binary:logistic objective function

I am working on a mortality prediction (binary outcome) problem with “base mortality probability” as my offset in the XGboost problem. I have used gbtree booster and binary:logistic objective function....
0
votes
0answers
31 views

what is the formula for r2 when using H20 GBM H2OGradientBoostingEstimator

I used a H2OGradientBoostingEstimator to do a classification (n features into a binary 0/1). What does the reported R^2 mean? Should I look more into AUC or into R^2? And for classification, what ...
0
votes
0answers
77 views

Variable importance in a GBM

I have build a model with a Gradient Boosting Machine (GBM) and calculated the feature importance. All features are factors. Now, I know which features are most important. However, the features have ...
2
votes
1answer
32 views

Caret: gradient vs gam boosting

What is the difference between a boosted additive model (e.g. caret model: gamboost) and a general stochastic gradient boosting model (caret model: gbm)? A gradient boosting model is additive by ...
1
vote
1answer
30 views

Updating decision tree for new data

Lets say you have trained a decision tree for 40 gigs of data. On Monday morning you receive 10Gig new data and produce some results quickly to report to your boss. Can you update the decision tree ...
4
votes
2answers
131 views

Is it ever recommended to use mean/multiple imputation when using tree-based predictive models?

Everytime that I am making some predictive model and I have missing data I impute categorical variables with something like "UNKNOWN" and numerical variables with some absurd number that will never be ...
0
votes
1answer
36 views

Defining the cases where Neural Networks outperform tree-based methods

It is well-known that neural networks are currently superior to most of the alternatives to do prediction from images (with CNNs) and sequential data (RNN, transformers...). However for other ...
2
votes
1answer
34 views

A regressor failed to learn extreme values

I am working on a regression problem using xgbclassifier (https://xgboost.readthedocs.io/en/latest/python/python_api.html) The output values range from 0 to 10 (log-normal distribution), but when I ...
2
votes
1answer
27 views

Why are AUC and logloss metrics not available in the “maximum metrics” table produced by H2O? [closed]

I am running the h2o.gbm algorithm using five-fold cross validation to predict a binary outcome. I want to see what threshold to use as a cutoff for classifying predictions, and I am wondering why the ...
0
votes
1answer
25 views

Gradient boosting understanding of residual picture

I recently looking at the Gradient boosting using following blog https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d I try to understand the picture but I need some help For ...
0
votes
1answer
38 views

How are the training and cross-validation metrics calculated in H2O?

I am working with the GBM algorithm in H2O in R. I am using 100% of the data as the training data, and then using 5-fold cross-validation to train and validate my model using 100% of the data. My ...
5
votes
2answers
157 views

Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
0
votes
1answer
50 views

Can AdaBoost be used for regression?

I know that AdaBoost can be used for classification, but how about regression? With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) ...
2
votes
1answer
294 views

Classification XGBoost vs Logistic Regression

I have a binary classification problem where the classes are slightly unbalanced 25%-75% distribution. I have a total of around 35 features after some feature engineering and the features I have are ...
0
votes
0answers
37 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
0
votes
0answers
85 views

Feature selection in xgboost vs GBM in H2O

I am working on a big data set( more than 100 variables) and 30 million observations. I tried to build 100 models with a grid search using both XGBoost and GBM in H2O (Sparkling Water). I realized ...
2
votes
1answer
76 views

How to interpret chart generated by gbm.perf function?

I'm new to GBM.Can you help me to understand the interpretation of gbm.perf function? I used following code in R best.iter = gbm.perf(train, method="cv") & got ...
0
votes
1answer
62 views

Are the same number of trees required to compare Random Forest against GBM?

My training set has 13,737 observations with 53 predictors. I need to compare the accuracy of Random Forest vs. GBM. For Random Forest, I set ntree = 128 [based on ...
0
votes
0answers
17 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
1
vote
0answers
25 views

XGBoost tree dump contains lots of empty trees

After fitting a regression model using XGBoost, I want to inspect the individual trees that were built. In the resulting table, I find a lot of 0-depth trees, i.e. trees with only a leaf node, and ...
1
vote
0answers
26 views

Evaluation of Classifier Performance on Imbalanced Dataset with Lift Chart

I trained a classifier on imbalanced dataset (label={0,1}) by assigning higher weight to rare event(label=1). Lift chart shows that the predicted and actual curves are very separated. I also trained ...
2
votes
1answer
18 views

binomial responses in h2o gbm

I am modeling the probability of success in a dataset where I have a both the number of trials and the number of successes (and, obviously, I am modeling $p_i=\frac{total successes}{total trials}$). I ...
2
votes
1answer
26 views

Learning Rate impact on model building time

I wanted to know that does learning rate impact the model building time in case of Gradient Boosted Trees. I do understand that increasing the number of trees have an impact( more the trees, more the ...
2
votes
2answers
54 views

Why are all predictions made by XGBoost distinct?

If I understood correctly the XGBoost is a framework that operates on gradient tree boosting. It means that behind the scenes, it uses a decision tree to make a prediction. So, from what I read in the ...
1
vote
1answer
14 views

Understanding `score` in LightGMB

I'm newly introduced to the LightGBM for a regression problem. Having read the documentation of LightGBM (here), I got puzzled about the ...
1
vote
0answers
51 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
0
votes
0answers
26 views

How do Gradient Boosted Trees calculate errors in classification? [duplicate]

I understand how gradient boosting works for regression when we build the next model on the residual error of the previous model - if we use for example linear regression then it will be the residual ...
2
votes
0answers
33 views

Implausible variable importance for GBM survival: constant difference in importance [closed]

I have a question about a GBM survival analysis. I'm trying to quantify variable importances for my variables (n=453), in a data set of 3614 individuals. The resulting graph wi th variable importances ...
0
votes
0answers
25 views

ada model- variables overall importance

I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees. I would like to have a ...
1
vote
1answer
22 views

Whats is the difference between using risk() and cvrisk() in the R package mboost

I am currently running an additive model using the function gamboost() in the package mboost. When using the ...
0
votes
0answers
33 views

Why AdaBoost works exactly the way it does

I understand the basic idea of AdaBoost -- when training weak classifiers, use more of the difficult examples. However, it puzzles me why I sould modify the weights the way AdaBoost does. There are, ...
0
votes
0answers
24 views

Conditions for Adaboost to perform well

Under which conditions does the AdaBoost algorithm yield good results even on weak learners (i.e. slightly better than random classifiers)?
1
vote
1answer
68 views

(Low cardinality) categorical features handling in gradient boosting libraries

In some popular gradient boosting libraries (lgb, catboost), they all seems like can handle categorical inputs by just specifying the column names of the categorical features, and pass it into a ...
1
vote
1answer
27 views

Differences between “in-bag” and “out-of-bag” empirical risks in the R package “mboost”

currently I am using the mboost R-package to estimate some additive models. When using the function gamboost(), you can control the hyper-parameters for boosting by using the option boost_control(). ...
0
votes
0answers
59 views

how to extract rules from final model made by caret

I have a made cross-validation (k=5) by caret package using C5.0 method. I have 21 features and 7000 instances. The C5.0 trials default is 40. The problem is C5.0 made > 1600 rules over 40 trials, ...
2
votes
1answer
38 views

How do I perform leave one out cross validation with boosting?

I'm working with the Anderson Iris data set and it is too small To split into a test and training set.I use boosting To make a classifier For determining the species Of flower Based on Variables in ...
1
vote
1answer
63 views

Why does GBM package make different predictions for the same data point (after factor issue is fixed)?

I tried to use a GBM model to make predictions for the same data point, but it gave me very different answers. Please see the example below. When using the entire dataset for predicting the first data ...
1
vote
1answer
44 views

How well gradient boosting can predict outside training values domain?

It has been said(link , link) that gradient boosting can predict values that fall outside of training domain for $Y$ in a regression problem. I intuitively sense that there is a distinction between ...
2
votes
1answer
91 views

Xgboost and repeated measures

I am learning xgboost and am planning on running a tree model. My dataset includes repeated measures. In a GLMM I would include the ID to account for repeated measures and I'm curious if I should do ...
2
votes
0answers
37 views

Applying boosting to predictions from a Random Forest

I have a class of datasets for a binary classification problem where it is known that Random Forest performs poorly compared with GBM or FFNN, rarely adding anything to an ensemble. I've had an idea ...
1
vote
0answers
18 views

Boosting using strong learners [duplicate]

At a high level, boosting is the process of adding many weak learners to form a strong learner. My professor had mentioned that in the case of boosting using regression trees, trees with a depth of ...
0
votes
0answers
20 views

Finding Null distribution for gbm interactions

I am trying to determine which interactions in a gbm model are significant using the method described in Friedman and Popescu 2008. My gbm is a classification model with 9 different classes. I'm ...
1
vote
1answer
26 views

How should I formulate the loss function/objective for this predictive modeling problem?

Let's say I have a big department store, selling all kinds of products, like clothing, shoes, cosmetics and electronics, etc. The data I have are daily sales by each item, like ...
0
votes
0answers
18 views

Predict malfunctionings with classifiers trained on truncated data?

I have an historical data-set of machines malfunctionings. I have data from different sensors, and a response variable of malfunctioning or not (1/0). I have difficulties in creating a classifier ...