Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

Way to stop model from overfitting in automated training pipeline?

I'm currently training a gradient boosting model for which I want to create an automated training pipeline containing hyperparameter optimization with hyperopt and also cross-validation. While trying ...
-1
votes
1answer
22 views

Identifying important features in forest model

I want to identify important features with sklearn for a random forest model and now I am not sure to use the train dataset or the entire population dataset for identification? The output from ...
0
votes
0answers
18 views

How to use interaction variables and indicator variable with GBM model in R? [closed]

I am new to Machine Learning and i have been working on a classification model which predicts donor is available or not. There is a variable which specifies whether user registered online voluntarily ...
0
votes
0answers
41 views

What to do when the meaning of a variable has changed over time?

I have dataset of a company with 2014 data with 15 variables then 2018 data with same 15 variables.I want to combine both the datasets however the meaning of 1 variable has changed meaning that ...
0
votes
1answer
23 views

Different machine learning models give contradictory results

I am relatively new to data analysis using machine learning and i got following problem while doing this machine learning problem. I have a data set with 80 features of 250 observations (rows) (but ...
0
votes
0answers
11 views

Boosting models on nested variables & imbalanced columns data frame

My data frame is defined by a structure such as below : The mainQ1 is the question stating do you have A,B,C,D,E,F and is the top of hierarchy with binary outcome ( 'Y' or 'N'). If it is answered 'Y'...
0
votes
0answers
14 views

Back out feature value to yield a certain response

Let's say we have a linear regression model $Y=\alpha+ \beta X_1$. It is easy to find the value of $X_1$ such that $Y=0$. It is $\frac{-\alpha}{\beta}$. Is there an analogue for nonparametric or ...
1
vote
1answer
24 views

why does random forest trees need to be deeper than gradient boosting trees

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF ...
1
vote
1answer
18 views

Model Size of Random Forest VS CatBoost

I trained models based on the same dataset, using random forest (sklearn) and CatBoost. I use n_estimators=1000 for random forest, and n_estimators(iterations)=1000 for CatBoost. The random forest ...
0
votes
2answers
59 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
1
vote
0answers
27 views

Difference between class_weight and scale_pos_weight in LightGBM

I have a very imbalanced dataset with the ratio of the positive samples to the negative samples being 1:496. The scoring metric is the f1 score,and my desired model is LightGBM. I am using the sklearn ...
0
votes
0answers
24 views

Gradient Boosting using Log-Likelihood Loss Function

I would love some help with gradient boosting using the negative log-likelihood as a loss function. According to a few sources, this should easily be possible. How would this gradient be calculated? ...
1
vote
1answer
25 views

Theory on custom loss functions for GBDT and other ML

I'm looking for resources on the theory behind choosing a loss function for ML---I'm interested in GBDT but for deep learning would work as well. I'd like to get a better understanding of how the loss ...
0
votes
0answers
19 views

Why are the predictions in gradient boosting for classification in terms of the log(odds)?

I'm trying to understand gradient boosting for BDTs using this video series: https://www.youtube.com/watch?v=StWY5QWMXCw I don't understand why the predicted values $F_m(x_i)$ at each step in the ...
1
vote
0answers
35 views

What derivative to use in Gradient boosting decision tree for a semi-supervised model

I am trying to build a semi-supervised prediction model with a Gradient Boosting decision trees. The learning phase is done using the following input: $X \in \mathbb{R}^{n\times p} $ $O(X) \in \...
0
votes
0answers
31 views

How to do variable selection for Gradient boosting models like Xgboost and LightGBM

I am building a classification model with about ~110 variables and that gave me an AUC of about 71.96 on validation. I added about 10 more features and my AUC value decreased to 71.56 (which led to ...
1
vote
1answer
62 views

Which method is correct for calculating total deviance explained for boosted regression trees?

I have the following output from a boosted regression trees model and I would like to calculate the total deviance explained. ...
0
votes
0answers
35 views

Controlling over-fitting in local cross-validation LightGBM

I am training a lightgbm model on a binary problem (~20% of events) with below parameters: ...
1
vote
1answer
38 views

Binary classification for imbalanced distribution of target/response class for age

I'm trying to build/train model that depends on many attributes where age is the most important one (it has significant impact on AUC). Overall target class count is quite balanced (+40% vs. -60%) ...
1
vote
0answers
16 views

How to combine from multiple probability in adaboost? [closed]

I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them? For simplicity, there ...
3
votes
1answer
31 views

Why does boosting accuracy change when run multiple times?

I have a set of 380 observations with 30 predictor variables and one response variable. I have determined using other methods (bagging and random forest) that there is a set of 7 variables that are ...
2
votes
0answers
24 views

What does “pairwise” distribuion in the R gbm package mean? [closed]

In the R gbm function (within the gbm R package), the distribution argument can get the value "pairwise". The help document states: I would like your help in understanding what is that "pairwise" ...
0
votes
0answers
20 views

Why we need to do bootstraping and extract confidence interval for classification problems?

My question is related to bootstraping and extracting confidence interval for classification problems. Let's say I have 25 number of data points and 2 features and use Gradient Boosting Machine (GBM) ...
2
votes
1answer
73 views

Negative Feature Importance Value in CatBoost LossFunctionChange

I am using CatBoost for ranking task. I am using QueryRMSE as my loss function. I notice for some features, the feature importance values are negative and I don't know how to interpret them. It says ...
1
vote
1answer
54 views

Regression when target has a wide range

I'm working on a regression model where I have to predict time. These times go from a few seconds to up to 30 min and more. I calculated the sMAPE through 1 minute bins of the target, and noticed ...
2
votes
0answers
23 views

In gradient boosting, what is the thing being boosted called?

Quoting from this answer that explains how to do boosting when a 'link' function is involved: "Instead, we can re-express this as a function of $L_i$, (in this case also known as the log odds) $$ \...
0
votes
0answers
57 views

Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]

I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
0
votes
2answers
36 views

why is number of epochs set as external parameter?

I am confused by the very notion of epochs in neural networks (as well as number of trees in gradient boosting). Gradient descent method (as most optimization algorithms) keep going until the loss ...
0
votes
0answers
25 views

How to convince people that developed predictive model based on Gradient Boosting Machine (GBM) has enough accuracy?

First of all, I'm not a data scientist. I'm an engineer that wants to use machine learning to do a binary classification based on a data that is extracted from computational modeling. I have four ...
2
votes
1answer
47 views

The error keeps decreasing with the increase of number of trees

I want to find optimized number of trees in Gradient Boosting. However, the error keeps decreasing with the increase of number of trees, I set the number of trees over 1000, but the error still keeps ...
1
vote
0answers
58 views

How does offset in XGBoost is handled in binary:logistic objective function

I am working on a mortality prediction (binary outcome) problem with “base mortality probability” as my offset in the XGboost problem. I have used gbtree booster and binary:logistic objective function....
0
votes
0answers
55 views

what is the formula for r2 when using H20 GBM H2OGradientBoostingEstimator

I used a H2OGradientBoostingEstimator to do a classification (n features into a binary 0/1). What does the reported R^2 mean? Should I look more into AUC or into R^2? And for classification, what ...
0
votes
0answers
149 views

Variable importance in a GBM

I have build a model with a Gradient Boosting Machine (GBM) and calculated the feature importance. All features are factors. Now, I know which features are most important. However, the features have ...
3
votes
1answer
76 views

Caret: gradient vs gam boosting

What is the difference between a boosted additive model (e.g. caret model: gamboost) and a general stochastic gradient boosting model (caret model: gbm)? A gradient boosting model is additive by ...
1
vote
1answer
76 views

Updating decision tree for new data

Lets say you have trained a decision tree for 40 gigs of data. On Monday morning you receive 10Gig new data and produce some results quickly to report to your boss. Can you update the decision tree ...
4
votes
2answers
197 views

Is it ever recommended to use mean/multiple imputation when using tree-based predictive models?

Everytime that I am making some predictive model and I have missing data I impute categorical variables with something like "UNKNOWN" and numerical variables with some absurd number that will never be ...
0
votes
1answer
44 views

Defining the cases where Neural Networks outperform tree-based methods

It is well-known that neural networks are currently superior to most of the alternatives to do prediction from images (with CNNs) and sequential data (RNN, transformers...). However for other ...
2
votes
1answer
39 views

A regressor failed to learn extreme values

I am working on a regression problem using xgbclassifier (https://xgboost.readthedocs.io/en/latest/python/python_api.html) The output values range from 0 to 10 (log-normal distribution), but when I ...
2
votes
1answer
41 views

Why are AUC and logloss metrics not available in the “maximum metrics” table produced by H2O? [closed]

I am running the h2o.gbm algorithm using five-fold cross validation to predict a binary outcome. I want to see what threshold to use as a cutoff for classifying predictions, and I am wondering why the ...
0
votes
1answer
28 views

Gradient boosting understanding of residual picture

I recently looking at the Gradient boosting using following blog https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d I try to understand the picture but I need some help For ...
0
votes
1answer
130 views

How are the training and cross-validation metrics calculated in H2O?

I am working with the GBM algorithm in H2O in R. I am using 100% of the data as the training data, and then using 5-fold cross-validation to train and validate my model using 100% of the data. My ...
5
votes
2answers
278 views

Which gamma regression model to use for extrapolation?

I'm looking for a regression model which would satify these requirements: My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have ...
0
votes
1answer
224 views

Can AdaBoost be used for regression?

I know that AdaBoost can be used for classification, but how about regression? With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) ...
1
vote
1answer
285 views

What is an intuitive interpretation of the leaf values in XGBoost base learners?

I'm learning XGBoost. The following is the code I used and below that is the tree #0 and #1 in the XGBoost model I built. I'm having a hard time understanding the meanings of the leaf values. Some ...
2
votes
1answer
1k views

Classification XGBoost vs Logistic Regression

I have a binary classification problem where the classes are slightly unbalanced 25%-75% distribution. I have a total of around 35 features after some feature engineering and the features I have are ...
0
votes
0answers
45 views

How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?

Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with $\...
0
votes
0answers
192 views

Feature selection in xgboost vs GBM in H2O

I am working on a big data set( more than 100 variables) and 30 million observations. I tried to build 100 models with a grid search using both XGBoost and GBM in H2O (Sparkling Water). I realized ...
2
votes
1answer
225 views

How to interpret chart generated by gbm.perf function?

I'm new to GBM.Can you help me to understand the interpretation of gbm.perf function? I used following code in R best.iter = gbm.perf(train, method="cv") & got ...
0
votes
1answer
67 views

Are the same number of trees required to compare Random Forest against GBM?

My training set has 13,737 observations with 53 predictors. I need to compare the accuracy of Random Forest vs. GBM. For Random Forest, I set ntree = 128 [based on ...
0
votes
0answers
20 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.