# Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

1,357 questions
Filter by
Sorted by
Tagged with
1 vote
19 views

### Build a born-again tree from XGboost set of trees

I am wondering whether an approach like what is described here: https://github.com/vidalt/BA-Trees for building a single tree from a Random Forest can be applied to an XGBoost model? Is there a ...
• 595
17 views

### Manual filtering is better than tree-based method

I am trying to solve a binary classification problem. The data is only slightly skewed (one class is 52-60% depending on the filtering), the dataset is small (so far) and consists of 4200 rows and ...
6 views

### biased estimates from xgboost [closed]

I have a data set which has 3802 elements. The data set have 7 variables including the target variable(severity). One dependent variable is "age". I apply to xgboost algorithm to the data ...
• 639
1 vote
22 views

### XGBoost multivariate time series 3 periods prediction

I've read a lot about using xgboost to forecast time series, but I feel like I've completly lost my mind and can't understand something very basic. Would like just a confirmation if my reasoning is ...
• 11
1 vote
16 views

### Pooling SHAP values from multiple imputed data

I have multiple imputed data and will be conducting an identical lightGBM model with the same input features in each of the imputed datasets. My aim is to calculate SHAP values (SHapley Additive ...
• 11
16 views

### Resizing input vectors of different lengths

I have a set of data taken from a number of machines. A simplified version of the dataset is shown below: ...
1 vote
22 views

### Curse of dimensionality using trees

The curse of dimensionality refers to the fact when a model tries to fit the data in a very high dimensional space (and there is not enough training data). In my mind, I believe that this curse ...
• 185
1 vote
15 views

### Oversampling for Continuous Values

I am trying to predict the processing time of a process by using xgboost regression algorithm in python. However I realised that my samples data is skewed to left and my algorithm struggles to predict ...
5 views

### Is scaling /missing value treatment required/useful in tree based models?

I'm working on a classification model using tree-based/boosting models. Does scaling/outlier treatment help improve the performance of tree-based models? How does it help (if it helps), why not (if it ...
• 203
6 views

### Score of LGBM Classifier ranging only between a short interval

I am working on a fraud problem and I am trying to predict either some market/stores has done fraudulent transactions or not. I've trained a boosting model (lgbm algorithm) on a unbalanced dataset. I'...
22 views

### Feature engineering to prepare an XGBoost classifier

Is there something in particular one should take into consideration when preparing data to a XGBoost Classifier? As it is tree based, it doesn't make any assumptions about data distribution like ...
• 137
1 vote
29 views

### Does evaluation metric not matter for training a model? And why?

I think I'm misunderstanding something with training ML tree models. There's a bunch of evaluation metrics we can use. It seems like this metric however is only used after training, which does not ...
• 111
20 views

### sample size justification in random forest or gradient boosting

I was wondering about the method of sample size calculation/power analysis in random forest and gradient boosting models. I already know the calculation of sample size using Gpower in regression model,...
• 21
1 vote
28 views

### Understanding feature importance for collinear features with tree-based models

I'm trying to understand how collinearity affects feature importance for tree-based models. My understanding is that tree-based models naturally overcome multicollinearity for the purposes of ...
14 views

### Should I Use rolling average for time series forecasting in this situation? (VAR, XGBoost regression, LSTM)

I am trying to perform multivariate time series forecasting using VAR, XGBoost regressor, and LSTM for comparison and I'm aware that it is common practice to smooth out the data by taking a rolling ...
23 views

### XGBoost Taylor Series approximation for los function gradient

Why do gradient boosted trees, including XGBoost, use the second order taylor approximation when computing gradients? I'm generally curious why this helpful and how it's easier than simply taking the ...
• 1,728
5 views

### Ensemble modeling strategy

Since the ensembling model requires the individual models to be different for effectiveness, can I run two xgb models with one model metric as recall and the other one's metric as precision and ...
• 203
1 vote
39 views

### Making sense of the Gain term in Gradient tree boosting

In the XGBoost Documentation they specify the Gain term as \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
• 115
1 vote
58 views

### xgboost and gridsearchcv in python

I have question about this tutorial. The author is doing hyper parameter tuning. The first window shows different values of hyperparameters Then he initializes ...
• 205
13 views

### tabular data to sequence regression

I am trying to map a tabular input input data [1x40] into a signal sequence [1x512] (e.g. sound) -regression task-. since my 40 input features are independent of each other and there is not a simple ...
15 views

### Unexplainable cyclical patterns on prediction intervals for time-series forecasting using Extreme Gradient Boosting regressor

I am following the documentation of skforecast to make time-series forecasting using the Extreme Gradient Boosting regressor (i.e., ...
• 1,097
26 views

### Different feature importance in different algorithms

I applied several ML algorithms to my data. My data is made of several predictive numeric features and a target categorical binary feature. My aim is to build a classifier and predict the target class,...
21 views

### What is the conclusion from cross validation scores?

I am training a model, and I'm using an xgboost model. I used the following piece of code to find the cross-validation score score=cross_val_score(final_classifier, X, y, cv=5, scoring="f1") ...
• 203
1 vote
19 views

### Xgboost algorithm tuning [closed]

I want to run an xgboost algorithm on my data. I have about 35 predictor features (numeric) and one target feature (catigorical). 1550 samples as rows. The training set contains about 1150 samples. ...
1 vote
47 views

### Gradient boosting on a loss/objective function without second derivatives

In principle, it should be possible to build a gradient boosted tree model on a loss function that only has (nonzero) first derivatives. I've found in practice xgboost and lightgbm make heavy use of ...
1 vote
50 views

### XGBoost poor calibration for binary classification on a dataset with high class imbalance

I've read a lot of threads/questions about this issue and I got conflicting answers. I've trained an XGBoost model on tabular data to predict the risk for a specific event (ie a binary classifier). ...
36 views

### Binary Classification using Machine Learning Models for longitudinal data in R

So I have longitudinal data with a binary target variable, and I'd like to perform binary classification using a random forest, xgboost, and glmnet (ridge/lasso/elastic net) model. Is this possible to ...
22 views

### How to compute the weighted class probability estimates in SAMME.R?

Does anybody know how to compute the weighted class probability estimates in the SAMME.R algorithm in ? I have created a simple example dataset as shown in the following. A decision tree with depth ...
381 views

### XGBoost when P>>N

Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were ...
• 155
9 views

### In boosting for regression trees, what is an example form of the final tree $\widehat{f}$?

In boosting for regression trees, the algorithm, for a given tree $b \in\{1,\ldots, B\}$ will fit a tree $\widehat{f^b}$ with $d$ splits to the training data $(X,r)$ where $r$ are the residuals. Then, ...
• 2,305
1 vote
32 views

### Giving more importance to under prediction (mean absolute error) than over prediction for forecasting

Just curious to hear any thoughts on weighting over prediction in mean absolute error to minimize the penalty since I'm more interested in under prediction, if that makes sense. Basically, I'm ...
• 13
152 views

### Is this a correct interpretation of percent importance?

In this scenario, the end goal is to determine which columns are most important. I use this term very very very loosely, because I know that interpretability is never straightforward but I just want ...
7 views

### Should the branch which has a negative gain be removed in XGBoost?

I have read that after building a regression tree to the max_depth, the algorithm starts to prune the branches which have a negative gain in a bottom-up manner. But ...
• 111
108 views

### Improving a forest model by dropping features below a percent importance threshold?

I'm wondering if there is a term for this process, where I can find more reading/information about it, and if this is valid or hacky overall. I've only used this process for tree-based models which ...
1 vote
18 views

### Is it disingenuous to test tree-based regressors on training data for the sake of comparison to a linear model?

I am currently performing an analysis in a domain that, in current literature, is predominantly populated by likelihood-based linear models (MLR, Poisson Regression, etc.). My intent is to implement ...
• 123
1 vote
11 views

### Leveraging hierarchical data in a GBM

A challenge that I encounter in a lot of modelling, is how to best handle hierarchical data for making prediction. A simple example of this is a basic (binary) classification problem, where I am ...
• 133
8 views

### How do I calculate the statistical power to find differences in the performance of the models to compare?

I am building an XG Boost model and I want to compare it with the results of a logistic model already built to predict hospitalization, however, I am asked to calculate the statistical power to find ...
• 1
1 vote
24 views

### Is there a formal methodology for controlling a variable in a tree based model?

I am currently building a xgboost model to classify data into 6 categories of risk for insurance policies. I have 5 years worth of policy holder data, including policy year. When building a GLM for a ...
• 11
1 vote
13 views

### What is the effect of grouping some of the target classes on Random Forest?

Suppose that the target variable consists of n distinct classes. Suppose also that the goal is to maximize the accuracy (F-score) of the specific class, say ...
• 63
23 views

### What’s the purpose of the 0.5 in the alpha term in Adaboost?

In the AdaBoost algorithm, the contribution of each weak learner is multiplied by a weight $\alpha$ given by $$\alpha_t = \frac{1}{2}ln\Big(\frac{1-\epsilon_t}{\epsilon_t}\Big)$$ In the Elements of ...
• 463
19 views

### Running model on full dataset or just test set?

I'm new to using XGBoost and I'm confused about how we should obtain the XGBoost predicted values for each data point. For example, the process for fitting and evaluating an XGBoost model is: ...
1 vote
25 views

### What are some good relative regression metrics?

Which relative regression metrics exist? What are their strengths and weaknesses? In what case do you use each? --> Bonus point if they are already/easily implemented in Scikit-Learn. I have a ...
40 views

### Imbalanced classification with xgboost in python with scale_pos_weight not working properly

I am using xgboost with python in order to perform a binary classification in which the class 0 appears roughly 9 times more frequently than class 1. I am of course using ...
• 171
12 views

### Is there a stochastic AdaBoost?

In sklearn, the AdaBoostClassifier and AdaBoostRegressor classes do not have the subsample and max_features parameters, which are responsible for the stochastic approach to building a boosting model. ...
11 views

### t-statistics in gradient boosted machine/forest such LightGBM

Is there a t-statistics in the gradient boosted forest regression model such as that in LightGBM? If so, how is it defined, extracted and used?
• 905
262 views

### XGBRegressor score (R2) vs. eval_metric (RMSE)

According to the API Reference, XGBRegressor().score() returns R2. However, according to the XGBoost Paramters page, the default ...
• 21
80 views

### Can Poisson deviance be used to evaluate models that use loss functions other than Poisson? (Such as MSE)

I am currently doing a a study on emergency department utilization rates at various geography levels. Especially of interest, are tree-based approaches to this analysis - namely random forest and GBMs....
• 123
84 views

### (XGBOOST) 5-fold cross validation test aucpr is always lower than train aucpr, is that overfitting?

I am using XGBOOST to construct a prediction model, but no matter what I do (including set gamma, subsample, eta), I will get results similar to the following picture. I did see all train and test ...