Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Can retraining predictive model solves Dataset shift?

Assuming we are using non-parametric models like gradient boosted tree, can retraining the model solves each of the dataset shift (1. covariate 2. prior probability 3. concept shift)?
0
votes
1answer
58 views
+100

LGBM binary classification probabilities do not match up to simulated expectations

I have a dataset with roughly 100 predictor variables and 20K examples that have binary classifications. My goal is to minimize binary cross-entropy loss, and given the nature of the data, I decided ...
0
votes
0answers
16 views

What are the disadvantages of the boosting method?

I was wondering if someone can please link me to an academic source that shows the disadvantages of the boosting method?
0
votes
0answers
25 views

Best Resource to learn about internal working of ML algorithms for an absolute beginner? [duplicate]

My previous question was about looking for pseudocodes for Boosting algorithms like XGB, Random Forests and LGBM. I figured it would be better if i had some resources to refer to, which would detail ...
0
votes
0answers
17 views

Do I need to select features for XGBoost [duplicate]

I have a regression task and I have around 79 numeric features that predict a numeric target value between 0 and 1, I used gradient boosting trees as there are several relationships both linear and ...
1
vote
1answer
65 views

Where can i find pseudocodes for tree ensemble Algorithms?

I would like to know if there is any such resource where we can get the pseudocodes for Gradient Boosting algorithms ? I am looking for pseudocodes for XGBoost, Random Forest and LGBM. I have read ...
0
votes
0answers
15 views

How to find the optimum split points for GBDT?

GBDT (Gradient Boosting Decision Tree) is an ensemble model of decision trees that are trained in sequence(i.e. an ensemble model of boosting). In each iteration, GBDT learns the decision tree by ...
0
votes
1answer
42 views

Forecasting sales

I got 95 weeks of sales data (i.e., 95 data points) for a retail business, whose plot looks like this: Sales are evidently seasonal. Also see plot for Year 1 against Year 2 Sales by Week of Year I ...
0
votes
0answers
17 views

Interpreting Variable Importance of XGBoost based decision tree models

While building an XGBoost based decision tree model, working with a set of say 100 variables gives a particular variable importance. Now if I want to improve performance through an additional set of ...
0
votes
0answers
24 views

restricting xgboost predictions value within a range

I am trying to predict the users rating on movies. These ratings are continuous ranging from 1 to 5. I have been using xgboost with objective function ...
1
vote
1answer
27 views

XGboost Regression Log Transform Target

I'm training a XGBoost Regressor to predict price which has a highly right skewed distribution. (all prices are positive) I took log transformation on the target thinking it would help to 1) stable ...
0
votes
1answer
19 views

Recommended Papers on GBM

I am writing my thesis and use a GBM to model insurance claim frequency. As I am currently writing the background, I am looking for good references on GBM. Do you have any recommendations for me? Any ...
0
votes
1answer
73 views

Machine learning, overfitting and artificial sample increments (LASSO, Boosting)

I am trying to use different types of Machine Learning (ML), LASSO, Elastic Net and Boosting, in a dataset with around 6000 observations and 120 regressors. To test the goodness-of-fit of results, I ...
1
vote
1answer
33 views

Is it ok to keep a very strong predictor and other weak predictors in the model? The model built is GBM

Age is coming out as a really strong predictor compared to other variables. This is a classification problem, the dependent variable is a (0/1)
1
vote
1answer
49 views

Interpretation of y-axis in partial dependence plot

First off, I know there are many questions on this site similar to this one. I've read them, and have not been able to find a solution. In Elements of Statistical Learning, the following figure shows ...
1
vote
1answer
30 views

Difference between GBTree and GBDart

From my understanding, GBDart drops trees in order to solve over-fitting. However, when tuning, using xgboost package, rate_drop, by default is 0. I understand this is a parameter to tune, however, ...
3
votes
1answer
32 views

GBM: does using a bag fraction of one pose any problems?

I am trying to fit a gbm model to some poisson distributed data and have run a cross validation scheme for some of gbm the parameters, including bag fraction. My results show that a bag fraction of ...
2
votes
1answer
71 views

Why are boosted trees difficult to interpret?

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble). ...
0
votes
1answer
24 views

How exactly should repeated measures situation be treated in machine learning models?

I was reading this SE about repeated measures and I was a little bit confused. I have a data set consisting of claim information for million locations for up to 5 years and 99.9% of the observations ...
0
votes
1answer
15 views

Does lightGBM requires the entire dataset in memory or Can it train the model by partially loading the data into memory? [closed]

I have a massive dataset -- ~120 M rows and 300 columns -- I want to use the entire dataset to train LightGBM. Is there a way to train it using chunks of data sequentially? If not what would you ...
2
votes
0answers
33 views

How to adjust for known prognostic clinical or molecular confounders when using survival analysis methods that are not based on Cox regression?

When using penalized Cox or Coxnet regression for survival analysis, it is possible to account and adjust for known prognostic clinical or molecular confounders by including them in your model as ...
0
votes
1answer
22 views

Closing prices are predicted very well but returns are predicted poorly

I'm learning some time series analysis and forecasting techniques, I've tried to predict stock prices for Netflix but I'm very confused. At first I've tried Auto ARIMA which gave me a straight line, ...
0
votes
0answers
13 views

Boosting Algorithm: What would happen if you omit the lambda (Make it equal 1 or make it too large) in this algorithm?

Taken from The Introduction to Statistical Learning textbook. I read the excerpt about boosting and have a fine conceptual understanding of the matter. Although I am curious why the learning parameter ...
0
votes
0answers
17 views

Are the predictions of gradient boosted trees made sequentially? If so, how do these algorithms not have slow prediction speed?

I understand the general process of how gradient boosted trees are trained. A sequence of learners is trained, with the initial learner being fit to the data, and each subsequent learner being fit to ...
0
votes
0answers
9 views

Why gradient boosting uses Taylor series for the classification problem

The value to determined for the leaf in gradient boosting given by optimising the value of gamma over the loss function. However this is done by using taylor series and dont understand why it is ...
2
votes
0answers
42 views

boosting and un-pruned decision tree

I have seen a lot of people claiming Boosting is used to reduce bias. Does it mean achieving high training accuracy, which could be achieved by un-pruned decision tree? Could anyone correct me where I ...
0
votes
0answers
15 views

How to account for temporal autocorrelation in BRT models?

I have GPS data (every 20 minutes) on storks flight heights and am modelling several predictors using Boosted Random Trees (using gbm and ...
2
votes
1answer
52 views

Multiclass gradient boosting: how to derive the initial guess, how to predict a probability

I have some questions regarding multiclass boosted-tree-algorithmus. Currently, I apply xgBoost as implemented in R to solve a multi-classification problem. According to StatQuest, for a simple two-...
0
votes
1answer
19 views

Caret GBM predict only produces 106 outputs while newdata has 403 rows… What's up?

I am training ML classifiers using caret to predict mortality in a clinical data set. Training with caret's gbm works well, but when I try to use predict, I get very strange results. Here is my code: ...
0
votes
1answer
19 views

Machine Learning models and “rare-event” independent variables

I have a data set which tries to predict a continuous variable, say house prices $Y$. My independent variables consits of things such as, square meters, number of bedrooms, bathrooms etc. However, I ...
0
votes
0answers
20 views

Tips/References For Building Boosting Trees Or Neural Networks With Large Datasets?

I have a lot of data that can be used to train a model - so much that I am not sure if my computer (16GB Ram) can handle all of the data at once. What are some ways to deal with this issue, given my ...
0
votes
1answer
15 views

How do you interpret your features when you standardize your data?

Let's say I have built a boosting tree or neural network and I standardized my features beforehand. When I built my model, I split my data into training, validation, and test sets - each with their ...
0
votes
0answers
24 views

Regression with zero-inflated outcome

I am trying to fit and tune a Regression gradient boosting model where my target variable is zero inflated (80% zero) and the rest of the values are distributed as positive and negative values (not ...
0
votes
0answers
25 views

Gradient Boosting vs Forward Stagewise Additive Model

Given that the famous Adaboost and Gradient Boosting are both some kind of approximation to Forward Stagewise Additive Modeling, why not directly fit a model using Forward Stagewise Additive Model? On ...
0
votes
0answers
26 views

Why gradient boosting use first-order Taylor expansion approximation?

The target of boosting at step $m$ is (see Wikipedia): $$F_{m}(x)=F_{m-1}(x)+\underset{h_{m} \in \mathcal{H}}{\arg \min }\left[\sum_{i=1}^{n} L\left(y_{i}, F_{m-1}\left(x_{i}\right)+h_{m}\left(x_{i}\...
2
votes
1answer
36 views

Predicted probabilities seem too low with Gradient Boosting Machine on `iris` data

I'm doing a test run of the Gradient Boosting Machine algorithm on the iris data with the caret package. ...
0
votes
0answers
14 views

Combine CatBoost with deep learning classifier

I'm using CatBoost to solve a binary classification problem. Most of my features are binary, but the order of features does matter. I've come up with a Recurrent Neural Network that has similar ...
0
votes
0answers
12 views

Splits in Decision Trees vs Dendrograms

gradient boosting is a supervised learning algorithm that splits/grows decision trees to improve predictions iteratively. hierarchical clustering is an unsupervised learning algorithm that splits/...
1
vote
0answers
23 views

Question for [Element of statistical Learning ] Page 357 [closed]

Here is the book link https://web.stanford.edu/~hastie/Papers/ESLII.pdf I am very confused about the statement here: I am familiar with CART and gradient boosting machine but I have no idea what we ...
0
votes
0answers
22 views

How to deal with unbalanced time series data for machine learning?

My understanding when it comes to unbalanced datasets is that we can randomly sample from the dominant class. What are some ways to deal with unbalanced data when we have time series data and the ...
2
votes
1answer
91 views

How does LightGBM deals with incremental learning (and concept drift)?

With some research I found that it updates the leaves (does not create new or remove old ones) is it right? How this happens? Another question is when the incremental learning is done in concept ...
1
vote
1answer
40 views

Steps in gradient boosting algorithm

Can some one please explain the 3rd step 2(c) in the below gradient boosting algorithm. I was under the impression, that the 2(c) computation is nothing but the mean of the corresponding terminal node ...
0
votes
0answers
18 views

How the first tree in gradient boosting classifier is constructed and the split criteria [duplicate]

I am aware how GB classifiers are constructed as regression trees and predictions are made, but not sure how the initial tree and node splitting for it is done. Can someone please explain how the ...
2
votes
1answer
34 views

Performance drops when adding a feature using XGBoost

I did some feature engineering with my data set. When I added on of the new features, the performance significantly dropped. How is this possible? I thought XGBoost is robust to irrelevant variables.
2
votes
0answers
28 views

Low OOB error but high CV error with MABoost

I am using Mirror Ascent Boosting (R package maboost) to learn a 3-class predictor over a set of 123 patients (very small , I know). Classes are almost balanced. I am getting excellent OOB errors (...
1
vote
1answer
60 views

int vs Float in regression modeling

This is general question to understand a concept. I have a dataframe with all columns having float values(precision varies from 2 to 8 digits). I use GBM to train my model. When i train my model ...
0
votes
0answers
33 views

How to compare feature selection regression-based algorithm with tree-based algorithms?

I'm trying to compare which feature selection model is more eficiente for a specific domain. Nowadays the state of the art in this domain (GWAS) is regression-based algorithms (LR, LMM, SAIGE, etc), ...
2
votes
1answer
99 views

catboost does not overfit - how is that possible?

I'm fitting and evaluating a CatBoostRegressor and a XGBRegressor to the same regression problem. I tried matching their ...
0
votes
0answers
19 views

Termination Condition for AdaBoost.R2

I can't quite wrap my head around the termination condition of AdaBoost.R2 as defined by Drucker in this paper. On page 2 of the paper he states to "repeat the following while the average loss* $\bar{...
1
vote
0answers
32 views

Decision tree- Alternative model to predict this data?

My data looks something this (for example): ...

1
2 3 4 5
16