Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Combining XGBoost and LightGBM

I'm working on a text classification problem and I am comparing LightGBM and XGBoost performances. Both on train and test sets I get roughly the same accuracy metrics, but what looks amusing to me is ...
0
votes
0answers
9 views

using the test population as an eval_set when doing hyperparameter optimization

I'm looking at this guide for hyperparameters optimization of boosting regressors using hyperopt. I noticed that for each trial, it uses the following code for the ...
1
vote
1answer
26 views

What is the difference between Gini index and Gini coefficient?

I am building a decision tree from scratch. I have been using entropy so far (calculated this way): ...
1
vote
0answers
21 views

“Jumping” among several interpolation techniques?

I am comparing several interpolation methods using monthly climatic data, through RMSE and a 10-fold cross-validation scheme. What I'm observing is that the performances vary from one month to ...
1
vote
1answer
30 views

The accumulative tree structure in a tree based gradient boosting

I'm playing with gradient boosting methods and with its python packages out there. I tried lightgbm, started with a high-dimensional input to predict a task. A left ...
0
votes
0answers
12 views

what's the split criteria used by catboost?

I'm trying to understand the split criteria used by catboost in the "plain" boosting mode (not interested in the "ordered" mode complication). In "algorithm 2 - Building a tree" they are saying that ...
0
votes
0answers
14 views

required sample size for establishing equivalence of a gradient boosting model on different population

I have a trained Gradient boosting Trees (regression) model with a given R2 metric (obtained via cross validation) Now I want to verify that the same model is valid for a very different population. Is ...
2
votes
1answer
50 views

Spelling out a detail in the gradient boosting machine algorithm for binary classification

This is a very long question, but perhaps people who are trying to deeply understand the Gradient Boosting Machine algorithm will think it's interesting. I've been working on understanding the ...
0
votes
0answers
21 views

Which is the best classification Algorithm to be used for finding the “second best class”?

I have a dataframe containing skillsets of players in different positions. I can build a classification problem for predicting the position of player based on the skillsets. However, the problem ...
0
votes
0answers
22 views

Estimate distribution from mean and prediction intervals

I'm using an ML-model (gradient boosting) to predict mean, upper and lower quantiles of a target variable which is gamma distributed. I want to construct distributions for the predictions and figured ...
1
vote
1answer
22 views

Comparison of regression models in terms of the importance of variables

I would like to compare models (multiple regression, LASSO, Ridge, GBM) in terms of the importance of variables. But I'm not sure if the procedure is correct, because the values ​​obtained are not on ...
3
votes
1answer
29 views

how to avoid overfitting in XGBoost model

I try to classify data from a dataset of 35K data point and 12 features Firstly i have divided the data into train and test data for cross validation After cross validation i have built a XGBoost ...
0
votes
0answers
15 views

Gradient boosting (GB) splitting methods (categorical features)

Regarding categorical features - ordinary trees treat categorical features in two main ways, CART - considers only binary splitting, those computing the mean response value (y_mean_i per each category ...
0
votes
0answers
15 views

how does using decision stump lead to an additive model?

In chapter 8 of ISLR it says boosting using stumps leads to an additive model. How would I derive $$f(X) = \sum^p_{j=1} f_j(X_j)$$ from $$\hat{f}(x) = \sum^B_{b=1} \lambda \hat{f}^b(x)$$?
0
votes
0answers
28 views

Why is the step length by default equal to 1 in gradient boosting?

On ESL p.359, it explains steepest descent: But in 10.37, it is trying to minimize the distance to g_im. It looks like the default step length is 1. Why is it so?
0
votes
0answers
42 views

Tuning threshold from multiclass ROC for Gradient Boosting Classifier?

I have created a ROC curve based on the output of a multiclass Gradient Boosting Classifier (See Figure below implemented from Yellowbrick ROCAUC: https://www.scikit-yb.org/en/latest/api/classifier/...
0
votes
0answers
10 views

What does “a distribution is consistent with a hypothesis class” mean?

What does "a distribution is consistent with a hypothesis class" mean? I came across the following statement in this pdf To see this, first note that for every distribution $P$ consistent with $...
3
votes
0answers
22 views

When should one use Bradley-Terry instead of gradient boosted trees for pairwise ranking

Both the Bradley-Terry model and Gradient boosted trees can be used to learn a ranking from pairwise comparisons (e.g. with libraries choix and XGboost). How do they relate to each other? Is there ...
3
votes
1answer
71 views

XGBOOST objective function derivation algebra

I need some help please with the derivation of xgboost objective function. I am following this online tutorial (Math behind GBM and XGBoost) How do you go from here $$ loss = \sum_{i=1}^{n} \left( ...
2
votes
1answer
70 views

Overfitting in extreme gradient boosting

My situation is: 36,197 observations/ 125 outcomes in training data 26 predictors A relatively successful prediction model has been built in a similar dataset using just logistic regression; I ...
2
votes
1answer
28 views

Calculate minimum accuracy for a boosting algorithm

Suppose, you are working on a binary classification problem. And there are 3 models each with 70% accuracy. If you want to ensemble these models using majority voting. What will be the minimum ...
2
votes
1answer
48 views

weak learning of 3-piece classifiers using decision stumps

I have a question about Example 10.1 in Shalev-Shwartz and Ben-David's "Understanding Machine Learning." The example means to illustrate weak learning of 3-piece classifiers $\mathcal H$ using ...
3
votes
0answers
52 views

How to explain random forest ML algorithm doesn't learn at all, while logistic regression learns very well?

My prediction task is as follows: Use name to predict people's ethnicity (into 4 categories: "English", "French", "Chinese", and "All others") as a multiclass classification problem. The name ...
1
vote
0answers
32 views

Calculate Gini Importance for Boosting Trees

From my understanding, Gini Importance means Mean Decrease in MSE for regression objectives, and Mean Decrease in Impurity for classification objectives. Typical random forest packages like ...
0
votes
0answers
10 views

Whats a good estimation for error measuremets when trying to predict values inside two bands?

I am using gradient boosting to predict two quantiles (upper and lower). The predicted value can be above, below, or in bounds. The problem I am facing is that counting the number of values in bound ...
0
votes
0answers
14 views

Mean Percentage Error on k-fold CV vs. Training Set

I'm performing a regression using a Gradient Boosting Machine. When comparing the cross-validation predictions with the true values, the Mean Percentage Error is around -6%. However, using the model ...
0
votes
0answers
37 views

Use of Random Forest in a paper

I am currently reading a paper from a Geophysics journal in which the authors apply a random forest to data sets from shear laboratory experiments. I am new to machine learning, and I'm confused about ...
0
votes
0answers
20 views

Combining features linearly before non-linear modelling

I want to construct a model like this: $$y = g(X, Z)$$ $$X = \alpha_1 x_1 + \alpha_2 x_2 + \dots + \alpha_n x_n$$ $$Z = \beta_1 z_1 + \beta_2 z_2 + \dots + \beta_n z_n$$ Where $x_i \text{ and } z_i$ ...
0
votes
0answers
25 views

GBM Models with Correlated Predictors

I am building GBM models in which Predictor A is more highly correlated with my response variable for certain values of A while Predictor B is more highly correlated with my response variable for ...
0
votes
1answer
55 views

Neural network (or deep learning) or gradient boosting for large data with a few features?

I'm dealing with a supervised regression problem, a financial data set with 5m observations and a few features (about 5 to 10). All the features are continuous numeric values, without any missing ...
0
votes
0answers
17 views

Selection of sample_weight for gradient boosted regression

I'm looking for any information on how the sample_weight parameter is typically selected for gradient boosted regression tree's - i.e. implementations such as ...
0
votes
0answers
20 views

Depth of trees in GBM: what does interaction.depth = 1 mean?

As far as I know, GBM does ensembling on weak learners which can be trees with one split. I have already read this post, but I could not find the answer for my question. My question is that what ...
0
votes
0answers
77 views

Can boosting be defined through a modification of weights in a binary classification problem?

Let's assume that we have a binary classification problem (our labels are only 0 and 1). We try to find a model that generates probabilities to observer 1. We measure quality of the model by log loss: ...
3
votes
1answer
149 views

Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to ...
0
votes
0answers
34 views

Non L2 loss-function in gradient boosting

As I understand the idea of gradient boosting in the (m+1)-th step we take the partial derivatives of the loss with respect to our new parameters $f^{[m]}(x^{(i)})$: $\tilde{y}^{(i)}=-\frac{\partial (\...
1
vote
0answers
18 views

Can we extend the XGBoost algorithm to higher order steps?

In XGBoost, the idea is at every round of boosting we add an additional model (a decision tree in XGBoost for trees). This model is learned to optimize the second order Taylor expansion of the loss of ...
1
vote
1answer
25 views

Decision trees - hypothesis test for quality of split

I was just introduced to the concept of decision tree. I read that hypothesis testing can be used to asses the quality of each split $$H_0: \text{split was bad}$$ $$H_a: \text{split was good}$$ I ...
1
vote
0answers
59 views

Violation of the IID assumption in Gradient Boosting

Generally, machine learning methods make little to no statistical assumptions. However, a key assumption they do make is that the data are IID. What are the implications of a violation of the ...
2
votes
1answer
30 views

Can additive boosting be used in case a classification?

In case of a regression we can apply a boosting approach as follows: Train a very simple model using the data set. Find a difference between the predictions and targets and use this difference as a ...
1
vote
1answer
64 views

Will Boosting reduce variance?

I've seen two conflicting arguments: In a Stanford cs229 note, the author claims that boosting will increase variance (see section 2.5): http://cs229.stanford.edu/notes/cs229-notes-ensemble.pdf Prof. ...
0
votes
0answers
24 views

Aggregating/Summing feature importance for categorical fields in xgboost

When I use a standard GBM package for categorical features, I receive importance metrics at that field level. For example: State: 0.75 However, when using XGB the importance is split by each of ...
2
votes
0answers
159 views

Learning to rank for time series with LightGBM data formatting

I have multiple time series, one for each item, say item1, item2,...itemN. N>=500. I have features associated for each of the items and a dependent variable. I am currently studying learning to rank ...
3
votes
1answer
105 views

How interaction terms are treated in Gradient Boosting?

In GAMs interaction terms have to be expressly specified as covariates, even for simple linear relationships. On the contrary, with Gradient boosting this is not nesessary because the algorithm itself ...
0
votes
1answer
66 views

Default threshold in cross-validation metrics - h2o R package

I created an cartesian grid of GBMs using h2o package in R and saved cross-validation metrics for each model in a data frame. So, for each model, I stored the ...
0
votes
0answers
13 views

Should we include “YEAR” column in time series data for any tree based algorithm? [duplicate]

I've a time series data having day, month, year and other columns. If I include year column during building random forest, then how can I predict for the new year. As the year passed won't be repeated ...
1
vote
1answer
23 views

First Iteration in Gradient Boosting Algorithm

For the input training set ${ \{ ({ x }_{ i }{ y }_{ i })\} }_{ i=1 }^{ n }$ if the loss function is L(y, f(x)), then we initialize the model $M_0$ by finding the $\gamma$ which minimizes: $$ F_0(x) ...
0
votes
0answers
23 views

Decision tree - least square empirical improvement

I'm looking for an explanation of formula (35) of the Gradient Boosting paper of Friedman [Friedman 2001, Greedy function approximation: a gradient boosting machine]. Here the least-squares ...
0
votes
1answer
22 views

Interpreting trees of XGBoostRegressor Model

I fitted a dmlc XGBoostRegressor model on a dataset with the variables ['CPI', 'Fuel_Price', 'Temperature', 'Unemployment'] and ...
1
vote
1answer
24 views

Why we fit xᵢ vs errorᵢ in Gradient Boosting

The basic idea of Boosting is to reduce bias by reducing training error in multiple iterations. However, I'm unable to understand how does combining multiple models which are trained by fitting ...
0
votes
0answers
29 views

Tips on improving random forest predictive accuracy when # of features is really low?

Working on a random forest predictive model with a continuous response variable and two continuous features. Normally when I do RF projects I use some sort of feature selection method to choose which ...