Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
1
vote
0answers
29 views

Does Wikipedia explain gradient boosting in wrong way?

Wikipedia's geral Gradient Boosting is: Friedman's Gradient Boosting is: Why wikipedia's gradient boosting fit h_m through pseudo-residuals while friedman uses line 4 to fit h? My question is not ...
2
votes
1answer
31 views

Why is sampling 50% of observations in stochastic gradient boosting equivalent to bootstrap sampling?

In the stochastic gradient boosting paper, Friedman (2002) writes that sampling half of the observations before each iteration is "roughly equivalent to drawing bootstrap samples at each ...
1
vote
0answers
14 views

Is the DART booster really equivalent to random forests?

Rashmi & Gilad-Bachrach (2015, p. 493) claim that if the dropout rate is 100%, i.e. if at each iteration the trees are grown completely independent of each other, the DART method is "no ...
0
votes
0answers
9 views

Improving robustness of XGBoost on large tabular dataset with small signal and lots of noise

I have been working with XGBoost on a large set of panel data. There are 20m+ rows with 200 features. The data includes weather related data points for 100s of cities, recorded every day, for several ...
0
votes
0answers
5 views

Partial_dependence_plot with gbt estimator has a mean response shift between curves computed by different methods ( 'brute' or 'recursion')

The new version of scikit-learn's partial_dependence function has the 'kind' additional option. With kind='average' one can compute the values for the partial dependence plot (PDP), with kind='...
0
votes
1answer
24 views

High Recall but too low Precision result in imbalanced data

I was training a model using XGBoost Classifier on heavy imbalanced data base with 232:1 of binary class. Because my training data contains 750k rows and 320 features (after doing many feature ...
0
votes
0answers
14 views

Gradient Boosting vs Gradient Descent

The section 10.10.2 of ESL claims that the difference between gradient boosting and steepest descent is that The tree components $t_{m} = T(x_{1};\theta_{m}),...,T(x_{N};\theta_{m})$ are not ...
0
votes
0answers
7 views

Why does XGBoost watchlist shows better metric scores than actual predictions?

So I have ran XGBoost in R some dozens of times now and always found it strange how when I set the "watchlist" parameter with test set, the metric scores it shows are always better than the ...
-1
votes
0answers
17 views

Assume a boosted classifier consists of weak hypotheses (aka. weak classifiers) that are each of them implemented by a threshold neuron. In that case, [closed]

Which of the following is True: Assume a boosted classifier consists of weak hypotheses (aka. weak classifiers) that are each of them implemented by a threshold neuron. In that case, the boosted ...
4
votes
1answer
47 views

Why Gradient Boosting is needed at all?

I am trying to learn from Hastie on boosting methods. A question has bothered me for weeks. The book describes Forward-Stagewise Fitting, that is, fit a weak learner to explain the residuals from the ...
3
votes
1answer
39 views

Does it make sense to do PCA before a Tree-Boosting model?

All I could find about this was this answer, which verifies my initial intuition that Decision trees, by virtue of doing recursive splitting of your samples, with splits being based on a single ...
1
vote
1answer
29 views

Are the important factors given by gradient boost more arbitrary compared to random forest

I compared the output from the two approaches using rfsrc() and gbm() in R respectively. The important factors given in the output from the two approaches are totally different. Since the importance ...
0
votes
1answer
22 views

Why does AdaBoost algorithm use weighted data points?

I am learning about AdaBoost algorithm. At each iteration, adaBoost set higher weight to mistake datapoints, and lower weight to correct classified data points. I do not understand why the algorithm ...
1
vote
1answer
55 views
+50

Correct approach to probability classification of a binary classifier

After having read a few articles and papers on probability calibration, I still don't have a clear understanding of what could be the best way to do model calibration in my case. I am using LightGBM (...
0
votes
0answers
17 views

can a Random Forest or Gradient Boosting Machine be used to solve a matching problem?

This is a totally made-up dataset, but this is the general idea (and yes, it's imperfect but that's not quite the point exactly). I have students and teachers, both entities have features. I have ...
1
vote
0answers
15 views

Is col sampling in GBM/XGB bias the variable importance?

GBM: Ensemble models like GBM, Light GBM or XGBoost Importance indicates how useful each feature was in the construction of the boosted decision trees within the model. The more an attribute is used ...
0
votes
0answers
14 views

Linear Models vs XGBoost

I am exploring the utility of logistic regression versus boosted tree methods (ie. XGBoost) in real world datasets. I am struggling to find situations (medical datasets) in which 1 model outperforms ...
1
vote
1answer
17 views

Do XBGoost and LightGBM only use trees as base learners?

From my understanding of GBM is that it can take not only decision trees as base learners, but different weak learners as well (e.g., linear models), since it relies on sequential gradient descent. ...
1
vote
0answers
21 views

what will be modified algorithm if this loss function will be modified? [closed]

I have a loss function of AdaBoost and $l(h(x),y)=e^{(-y*h(x))}$ and what will be modified algorithm if we modify loss function to $l(h(x),y)=ln(1+e^{(-y*h(x)})$. I was not able to find a way to ...
1
vote
0answers
60 views
+50

LightGBM model improvement when the focus is on probability prediction

I am building a binary classifier using LightGBM. The goal is not to predict the outcome as such, but rather to predict the probability of the target even. To be more specific, it's more about ranking ...
0
votes
0answers
21 views

D-Calibration for Survival Models with imbalanced outcomes

I am building an accelerated time failure (AFT) model on a highly imbalanced data set 90% survival 10% death. I understand that we can not use Brier score because of the outcome imbalance, Brier score ...
0
votes
0answers
7 views

Best Way To Encode Categorical Variable To Capture Impact of Each Unique Value For Tree Based Model When Data has Collinearity

I'm working on a project right now where I'm looking to use XGBoost to model a binary classification problem and use feature importances to look at the relative importances of group characteristics in ...
0
votes
0answers
12 views

How should I change boosting hyperparameters as signal/noise ratio decreases?

I am trying to get some intuition for how various boosting hyperparameters should be tweaked based on the underlying DGP (for concreteness, I have been working with XGBoost, whose hyperparameters are ...
2
votes
2answers
26 views

Question on loss function notation in Elements of Statistical Learning II

In Elements of Statistical Learning II on page 349, the multinomial deviance loss function is given by $L(y,p(x))=-\sum_{k=1}^KI(y=G_k)f_k(x)+\log(\sum_{\ell=1}^Ke^{f_\ell(x)})$, but there is no ...
0
votes
1answer
10 views

Problem with Clamping mask in Biomod2

I'm trying to project an alien species distribution with Biomod2, algorithm GBM, random pseudo-absences in equal number to presences, background restricted to the zoogeographic realms in which the ...
0
votes
0answers
8 views

Best Way to Report Calibration

I'm wondering what is the most accepted way to report metrics for model calibration. I have seen R2, slope and the brier score being used. In addition, are there any packages in Python that can be ...
2
votes
2answers
37 views

Does Adaboost ensemble use bootstrapping?

I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as I understand the general idea behind it: Give more weight to ...
6
votes
2answers
78 views

What to do when your training and testing data have different distributions

I am training a XGBoost regression model for predicting number of applications and the range of the target variable in train and test data set is different. For e.g: In Train data : Minimum ...
1
vote
0answers
29 views

Why is there no improvement when training Xgboost with pseudo-Huber loss?

In this StackOverflow post I asked if there was something wrong with my syntax when training an XGboost model (in R) with the native pseudo-Huber loss ...
1
vote
0answers
13 views

How to model a new short term beaviour in time series using ML approaches?

I had been modeling time series which contain many SKUs , due to corona there was a short-term drift in seasonality. I want to know how can i deal with this problem. Usually, sales are high in April, ...
2
votes
0answers
8 views

Gradient Boosting algorithm

What is beta for? Why not just fit h_m (get a_m) without this beta? since h_m is an estimator for pseudo-residuals
3
votes
0answers
47 views

Xgboost: does data need to be standardized when using shrinkage (parameters lambda, gamma)?

In this similar question about the implications of standardizing the features of data, the answer is that it is not important. However, (as pointed out in the comments on that post) I am interested in ...
0
votes
0answers
16 views

Huge AUC difference between using predictions from xgboost.train and XGBClassifier

I'm using two different versions of XGBoost modeling, and seeing that the two versions are producing vastly different AUC results. As far as I know, the XGBClassifier.fit() method should be using the ...
0
votes
0answers
7 views

R: Evaluate Gradient Boosting Machines (GBM) for Regression

Which are the best metrics to evaluate the fit of a GBM algorithm in R (metrics, graphs, ratios)? And how interpret them?
2
votes
1answer
33 views

How can I directly compare GBM and XGBoost?

I'm struggling to wrap my head around how GBM and XGBoost are mathematically related. Specifically, I don't know how to relate the steps described in the XGBoost vignette to the steps of the gradient ...
0
votes
0answers
12 views

What is actually returned by an AdaBoost classifier?

I'm trying to implement my own AdaBoost classifier using decision trees. I understand how the algorithm uses the different models generated and their weights to determine what the predicted label is. ...
0
votes
0answers
10 views

XGBoost and small data - rule of thumb

What is the rule of thumb for XGBoost (tree-based regression) max_depth in small training samples (~5K rows)? Also, what's the max. number of predictors that one ...
2
votes
1answer
38 views

LightGBM tree complexity optimization

We're using LightGBM on a dataset with a low signal-to-noise ratio (very easy to overfit, achieving OOS accuracy of 60% is considered a big win) where most features have low predictive power and the ...
1
vote
0answers
8 views

Online Tree Based Algorithms

Linear regression and logistic regression can do online training(i.e. continuous training as new data arrives) via stochastic gradient descent. Are there any tree based algorithms which can ...
0
votes
1answer
26 views

XGBoost, Imbalanced Data and CalibratedClassifierCV

I am currently working with a slightly imbalanced dataset (9% positive outcome) and am using XGBoost to train a predictive model. ...
0
votes
0answers
10 views

Boosting classification trees

After reading a paper about boosting regression tres, which follow the next algorithm: (B is the number of trees that are built) I wonder how the above steps might change for classification trees?. ...
0
votes
0answers
23 views

minimum sample size for xgboost for regression

I have about 6 variables (4 are numeric and 2 of them are binary) and 1000 observations to predict a count variable ( number of times an item has been purchased). Can I use xgboost model to train on ...
1
vote
1answer
52 views

XGBoost Calibration

I have an imbalanced dataset and am using XGBoost to create a predictive model. xgb = XGBClassifier(scale_pos_weight = 10, reg_alpha = 1) Although my recall ...
0
votes
0answers
26 views

On the selection of predictor for machine learning methods

I have a large number of predictors and I'm using two machine learning algorithms random forests and boosted regression trees. I thought of filtering the variables and keep only the most important ...
1
vote
1answer
29 views

Which loss functions does h2o.gbm use by default?

The GBM implementation of the h2o package only allows the user to specify a loss function via the distribution argument, which defaults to ...
0
votes
0answers
16 views

Using Topic Modeling for Text Classification

I'm trying to use topic modeling using Latent Dirichlet Allocation as input for text classification problem. Although, I'm not getting good results by doing this. The data has three variables doc id, ...
0
votes
0answers
6 views

Many identical false negatives in multi-output classification

I used an XGB model to classify 12 categories (I call them classes 1-12). I've found that a particular false positive, 5, is predicted frequently for what should be '2', according to the actuals that ...
2
votes
1answer
41 views

Should the day of the week variable be splitted into 7 columns or is one enough? Time-series forecast

I have time-series data, in which I added a variable "dayofweek" with the day of the week varying from $0$ (Monday) to $6$ (Sunday) (Python's default). I'm using boosting models like GBM and ...
1
vote
0answers
21 views

Regarding bagging, boosting and the NFL theorem

According to my understanding, bagging and boosting work in the following way: Bagging: Combine several high-variance/low-bias models to produce an ensemble model with lower variance and equal bias. ...
1
vote
0answers
48 views

How to simply understand gradient boosting on ranking problem?

I am reading Chris Burge's paper about LambdaRank, LambdaMART for learning to rank. We only need to compute the lambda, which is relevant to gradients, and use it to update model parameters, no need ...

1
2 3 4 5
25