Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
1 vote
1 answer
18 views

Build a born-again tree from XGboost set of trees

I am wondering whether an approach like what is described here: https://github.com/vidalt/BA-Trees for building a single tree from a Random Forest can be applied to an XGBoost model? Is there a ...
  • 595
0 votes
0 answers
17 views

Manual filtering is better than tree-based method

I am trying to solve a binary classification problem. The data is only slightly skewed (one class is 52-60% depending on the filtering), the dataset is small (so far) and consists of 4200 rows and ...
0 votes
0 answers
6 views

biased estimates from xgboost [closed]

I have a data set which has 3802 elements. The data set have 7 variables including the target variable(severity). One dependent variable is "age". I apply to xgboost algorithm to the data ...
  • 639
1 vote
0 answers
22 views

XGBoost multivariate time series 3 periods prediction

I've read a lot about using xgboost to forecast time series, but I feel like I've completly lost my mind and can't understand something very basic. Would like just a confirmation if my reasoning is ...
  • 11
1 vote
0 answers
16 views

Pooling SHAP values from multiple imputed data

I have multiple imputed data and will be conducting an identical lightGBM model with the same input features in each of the imputed datasets. My aim is to calculate SHAP values (SHapley Additive ...
  • 11
2 votes
1 answer
16 views

Resizing input vectors of different lengths

I have a set of data taken from a number of machines. A simplified version of the dataset is shown below: ...
1 vote
1 answer
22 views

Curse of dimensionality using trees

The curse of dimensionality refers to the fact when a model tries to fit the data in a very high dimensional space (and there is not enough training data). In my mind, I believe that this curse ...
  • 185
1 vote
1 answer
15 views

Oversampling for Continuous Values

I am trying to predict the processing time of a process by using xgboost regression algorithm in python. However I realised that my samples data is skewed to left and my algorithm struggles to predict ...
0 votes
0 answers
5 views

Is scaling /missing value treatment required/useful in tree based models?

I'm working on a classification model using tree-based/boosting models. Does scaling/outlier treatment help improve the performance of tree-based models? How does it help (if it helps), why not (if it ...
  • 203
0 votes
0 answers
6 views

Score of LGBM Classifier ranging only between a short interval

I am working on a fraud problem and I am trying to predict either some market/stores has done fraudulent transactions or not. I've trained a boosting model (lgbm algorithm) on a unbalanced dataset. I'...
0 votes
0 answers
22 views

Feature engineering to prepare an XGBoost classifier

Is there something in particular one should take into consideration when preparing data to a XGBoost Classifier? As it is tree based, it doesn't make any assumptions about data distribution like ...
  • 137
1 vote
0 answers
29 views

Does evaluation metric not matter for training a model? And why?

I think I'm misunderstanding something with training ML tree models. There's a bunch of evaluation metrics we can use. It seems like this metric however is only used after training, which does not ...
  • 111
2 votes
0 answers
20 views

sample size justification in random forest or gradient boosting

I was wondering about the method of sample size calculation/power analysis in random forest and gradient boosting models. I already know the calculation of sample size using Gpower in regression model,...
  • 21
1 vote
0 answers
28 views

Understanding feature importance for collinear features with tree-based models

I'm trying to understand how collinearity affects feature importance for tree-based models. My understanding is that tree-based models naturally overcome multicollinearity for the purposes of ...
0 votes
0 answers
14 views

Should I Use rolling average for time series forecasting in this situation? (VAR, XGBoost regression, LSTM)

I am trying to perform multivariate time series forecasting using VAR, XGBoost regressor, and LSTM for comparison and I'm aware that it is common practice to smooth out the data by taking a rolling ...
  • 1
0 votes
0 answers
23 views

XGBoost Taylor Series approximation for los function gradient

Why do gradient boosted trees, including XGBoost, use the second order taylor approximation when computing gradients? I'm generally curious why this helpful and how it's easier than simply taking the ...
  • 1,728
0 votes
0 answers
5 views

Ensemble modeling strategy

Since the ensembling model requires the individual models to be different for effectiveness, can I run two xgb models with one model metric as recall and the other one's metric as precision and ...
  • 203
1 vote
1 answer
39 views

Making sense of the Gain term in Gradient tree boosting

In the XGBoost Documentation they specify the Gain term as \begin{equation} Gain=\frac{1}{2} \left[ \frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda}- \frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\...
  • 115
1 vote
1 answer
58 views

xgboost and gridsearchcv in python

I have question about this tutorial. The author is doing hyper parameter tuning. The first window shows different values of hyperparameters Then he initializes ...
0 votes
0 answers
13 views

tabular data to sequence regression

I am trying to map a tabular input input data [1x40] into a signal sequence [1x512] (e.g. sound) -regression task-. since my 40 input features are independent of each other and there is not a simple ...
0 votes
0 answers
15 views

Unexplainable cyclical patterns on prediction intervals for time-series forecasting using Extreme Gradient Boosting regressor

I am following the documentation of skforecast to make time-series forecasting using the Extreme Gradient Boosting regressor (i.e., ...
  • 1,097
0 votes
1 answer
26 views

Different feature importance in different algorithms

I applied several ML algorithms to my data. My data is made of several predictive numeric features and a target categorical binary feature. My aim is to build a classifier and predict the target class,...
0 votes
0 answers
21 views

What is the conclusion from cross validation scores?

I am training a model, and I'm using an xgboost model. I used the following piece of code to find the cross-validation score score=cross_val_score(final_classifier, X, y, cv=5, scoring="f1") ...
  • 203
1 vote
0 answers
19 views

Xgboost algorithm tuning [closed]

I want to run an xgboost algorithm on my data. I have about 35 predictor features (numeric) and one target feature (catigorical). 1550 samples as rows. The training set contains about 1150 samples. ...
1 vote
1 answer
47 views

Gradient boosting on a loss/objective function without second derivatives

In principle, it should be possible to build a gradient boosted tree model on a loss function that only has (nonzero) first derivatives. I've found in practice xgboost and lightgbm make heavy use of ...
1 vote
1 answer
50 views

XGBoost poor calibration for binary classification on a dataset with high class imbalance

I've read a lot of threads/questions about this issue and I got conflicting answers. I've trained an XGBoost model on tabular data to predict the risk for a specific event (ie a binary classifier). ...
0 votes
0 answers
35 views

Binary Classification using Machine Learning Models for longitudinal data in R

So I have longitudinal data with a binary target variable, and I'd like to perform binary classification using a random forest, xgboost, and glmnet (ridge/lasso/elastic net) model. Is this possible to ...
0 votes
0 answers
22 views

How to compute the weighted class probability estimates in SAMME.R?

Does anybody know how to compute the weighted class probability estimates in the SAMME.R algorithm in [1]? I have created a simple example dataset as shown in the following. A decision tree with depth ...
5 votes
1 answer
381 views

XGBoost when P>>N

Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were ...
  • 155
0 votes
0 answers
9 views

In boosting for regression trees, what is an example form of the final tree $\widehat{f}$?

In boosting for regression trees, the algorithm, for a given tree $b \in\{1,\ldots, B\}$ will fit a tree $\widehat{f^b}$ with $d$ splits to the training data $(X,r)$ where $r$ are the residuals. Then, ...
  • 2,305
1 vote
2 answers
31 views

Giving more importance to under prediction (mean absolute error) than over prediction for forecasting

Just curious to hear any thoughts on weighting over prediction in mean absolute error to minimize the penalty since I'm more interested in under prediction, if that makes sense. Basically, I'm ...
3 votes
2 answers
152 views

Is this a correct interpretation of percent importance?

In this scenario, the end goal is to determine which columns are most important. I use this term very very very loosely, because I know that interpretability is never straightforward but I just want ...
0 votes
0 answers
7 views

Should the branch which has a negative gain be removed in XGBoost?

I have read that after building a regression tree to the max_depth, the algorithm starts to prune the branches which have a negative gain in a bottom-up manner. But ...
  • 111
3 votes
1 answer
107 views

Improving a forest model by dropping features below a percent importance threshold?

I'm wondering if there is a term for this process, where I can find more reading/information about it, and if this is valid or hacky overall. I've only used this process for tree-based models which ...
1 vote
1 answer
18 views

Is it disingenuous to test tree-based regressors on training data for the sake of comparison to a linear model?

I am currently performing an analysis in a domain that, in current literature, is predominantly populated by likelihood-based linear models (MLR, Poisson Regression, etc.). My intent is to implement ...
  • 123
1 vote
0 answers
11 views

Leveraging hierarchical data in a GBM

A challenge that I encounter in a lot of modelling, is how to best handle hierarchical data for making prediction. A simple example of this is a basic (binary) classification problem, where I am ...
  • 133
0 votes
0 answers
8 views

How do I calculate the statistical power to find differences in the performance of the models to compare?

I am building an XG Boost model and I want to compare it with the results of a logistic model already built to predict hospitalization, however, I am asked to calculate the statistical power to find ...
  • 1
1 vote
0 answers
24 views

Is there a formal methodology for controlling a variable in a tree based model?

I am currently building a xgboost model to classify data into 6 categories of risk for insurance policies. I have 5 years worth of policy holder data, including policy year. When building a GLM for a ...
  • 11
1 vote
0 answers
13 views

What is the effect of grouping some of the target classes on Random Forest?

Suppose that the target variable consists of n distinct classes. Suppose also that the goal is to maximize the accuracy (F-score) of the specific class, say ...
  • 63
0 votes
0 answers
23 views

What’s the purpose of the 0.5 in the alpha term in Adaboost?

In the AdaBoost algorithm, the contribution of each weak learner is multiplied by a weight $\alpha$ given by $$ \alpha_t = \frac{1}{2}ln\Big(\frac{1-\epsilon_t}{\epsilon_t}\Big) $$ In the Elements of ...
0 votes
0 answers
19 views

Running model on full dataset or just test set?

I'm new to using XGBoost and I'm confused about how we should obtain the XGBoost predicted values for each data point. For example, the process for fitting and evaluating an XGBoost model is: ...
1 vote
0 answers
25 views

What are some good relative regression metrics?

Which relative regression metrics exist? What are their strengths and weaknesses? In what case do you use each? --> Bonus point if they are already/easily implemented in Scikit-Learn. I have a ...
0 votes
0 answers
39 views

Imbalanced classification with xgboost in python with scale_pos_weight not working properly

I am using xgboost with python in order to perform a binary classification in which the class 0 appears roughly 9 times more frequently than class 1. I am of course using ...
  • 171
0 votes
0 answers
12 views

Is there a stochastic AdaBoost?

In sklearn, the AdaBoostClassifier and AdaBoostRegressor classes do not have the subsample and max_features parameters, which are responsible for the stochastic approach to building a boosting model. ...
0 votes
0 answers
11 views

t-statistics in gradient boosted machine/forest such LightGBM

Is there a t-statistics in the gradient boosted forest regression model such as that in LightGBM? If so, how is it defined, extracted and used?
  • 905
2 votes
2 answers
260 views

XGBRegressor score (R2) vs. eval_metric (RMSE)

According to the API Reference, XGBRegressor().score() returns R2. However, according to the XGBoost Paramters page, the default ...
  • 21
2 votes
1 answer
80 views

Can Poisson deviance be used to evaluate models that use loss functions other than Poisson? (Such as MSE)

I am currently doing a a study on emergency department utilization rates at various geography levels. Especially of interest, are tree-based approaches to this analysis - namely random forest and GBMs....
  • 123
0 votes
0 answers
84 views

(XGBOOST) 5-fold cross validation test aucpr is always lower than train aucpr, is that overfitting?

I am using XGBOOST to construct a prediction model, but no matter what I do (including set gamma, subsample, eta), I will get results similar to the following picture. I did see all train and test ...
0 votes
0 answers
96 views

which dataset to send as eval set in xgboost, catboost, and etc, when using optuna

In some boost models there are option to send eval set while fitting the model. for example: ...
  • 77
1 vote
1 answer
39 views

Training discrete time hazard model with xgboost on right censored loan data

I am currently developing a loan default risk model using a discrete time hazard approach with xgboost. The goal is to generate a series of predicted monthly default probabilities using a new ...

1
2 3 4 5
28