# Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

1,445 questions
Filter by
Sorted by
Tagged with
1 vote
23 views

### Partial derivative notation extreme gradient boosting

What is meant by this notation regarding gradient boosting source: $$g_i = \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)})$$ Is it the partial derivative of $l$ w.r.t. $\hat{y}_i^{(t-1)}$ , i....
• 165
94 views

### How to interpret the deviance plot by boosting models

This plot is taken from a gradient boosting regression example in the scikit-learn documentation. What does deviance mean? How should this plot be interpreted? In which case do we have over/...
13 views

### XGBoost's subsample = 0?

In my use of XGBoost with the gradient-based method, I inadvertently set subsample to 0, yet it surprisingly returned a good result. I am not sure how to explain it well. Any idea from the community?
1 vote
117 views

### Help with Classification model for S&P500

I have started a project in order to develop my coding skills, where I am predicting next month's S&P500 return direction based on some macroeconomic and financial variables. These datasets have ...
• 13
33 views

### Ordinal vs multinominal classification in XGboost: differences in one-hot encoding

I have followed this post and tried to see if there will be any difference in predicted probabilities if I use different one-hot encoding in XGboost. This is my code with some dummy data, which is ...
• 221
112 views

### Ordinal log-loss in a multiclass classification in XGBoost?

I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High). What I would ...
• 221
1 vote
94 views

### Multivariate Time Series dataset preparation

I am a bit confused with the time series dataset preparation. From the internet, I saw all examples which used tree-based models, had input features and target defined as: ...
• 61
34 views

### Cross-validation and automated binning of a continuous variable for a continuous target

I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
20 views

### Best practices when training an xgboost model as part of a larger model?

I would like to train a model end-to-end that uses the output from an xgboost model as an input. I've successfully implemented full-batch gradient descent into my pipeline with jax, following this ...
1 vote
38 views

### Assigning class weights during XGBoost training phase [closed]

I am using the following code to train a XGBoost model. I have specified the weights in random_search fit. My question is, does this adjust the weights during training fit or should I include "...
• 11
1 vote
75 views

### How to use priors to impute values at an individual level and replicate a distribution of the population?

I am trying to correct a variable from a survey that has measurement error. To do this, I have been taking this column as if it was missing and imputing new values based on the predictions of an ...
1 vote
64 views

### XGBoost - Linear Tree

I’ve been reading about linear tree models particularly the linear-tree package and the option to use linear trees in LightGBM if one sets the parameter linear_tree ...
• 11
30 views

### Why are SHAP interaction values inconsistent with those from a linear model?

I have fit a gradient boosted regressor to some tabular data. When looking at dependence plots there are some clear interaction effects e.g. effect of gender on the outcome measure reverses dependent ...
• 233
1 vote
9 views

### Avoiding over-fitting in Gradient Boosted tree models when multiple sequential observations share the same label [closed]

I am trying to train a Multi-class classification model where every K minutes I receive a set of features x and use the set to ...
48 views

### XGBoost Calibration for weighted loss function

I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
20 views

### random forest/ boosting and overfitting issue

I guess we can use CV to optimize how many trees (i.e. ntree or nround for boosting) in order to not overfit. It's like one of the way to not overfit. If we have too many complex trees that we learned ...
20 views

### Why my XGBoostClassifier model results in perfect accuracy despite dropping corelated features?

I am trying to do a binary classification on ticket canceling data from kaggle. I know this question has been asked before. For example here and here Summary of what I learned in those references: ...
• 101
46 views

### XGBoost imbalanced class problem

I am currently working on a personal project for predicting financial equities movements into 3 classes. The three classes are 1.) above a certain percentage move within a certain timeframe, 2.) below ...
16 views

### Update/ recalibrate XG Boost, Random Forest, GLM models for external validation

I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects. I have ...
• 11
89 views

### Monotone constraints in decision tree regressor or random forest regression

after I've spent several weeks trying to fit a regression model to my flood damage data (x1=water height, x2=adaptation height, x3=(x1-x2), y=damage), it is now time for my very first question on ...
64 views

For a set of features {a, b, c, d . . . n}, XGBoost can easily learn, say, a*d. In practice can it also effectively learn a/c? Or (a + b + c + 2)/d? Or (c^(2d))/(b^a)? I'd imagine some of this depends ...
• 161
1 vote
52 views

### Correct probability with XGBoost in R

It seems that I don't understand results of XGBoost. I've made a model with one variable and a binary target variable, and only one tree. Sample of data structure is presented below. ...
• 13
1 vote
58 views

### Understanding a calibration plot for lightGBM binary classifier

I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
65 views

### Trouble Achieving Optimal Parameters for XGBoost Regressor with Sin(x) Time Series

I'm facing a challenge while attempting to optimize parameters for an XGBoost regressor (Python) using time series data. I've created a time series using the following code: ...
26 views

### What algorithm/approach is best for a multiclass classifier where there are a significant amount of misclassifications in the source data?

I am using document embeddings (100 dimensions) to train a classifier on text data, however, I am getting poor results which I am attributing to the fact that there are a large proportion of ...
• 31
1 vote
63 views

### Unusual results from XGBoost learning curves

I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
• 21
37 views

### Can I add the gain feature importance metric for multiple features from XGBoost?

I would like to combine the "gain" feature importance metric from multiple features since this is potentially more informative. In my example, I would like to compare the importance of both ...
36 views

### Why is the initial prediction of gradient boosting classifier based on log of odds? Why can't we work with probabilities instead?

I am unable to understand the role of log odds in GBM classification. Why can't we just work with probabilities?
16 views

### Working with subsets of values from single category in XGBoost

Since version 1.5, XGBoost supports categorical data out of the box, which is a convenient way to skip the one-hot pre-processing step and allow for if X in values ...
57 views

### Double Machine Learning: What kind of "naive" estimator do the authors use to get such a bias?

I am reading the double-machine learning paper by Chernozhukov et al. (2018), in Example 1.1. the authors consider the partially linear model: Y = D \theta_0 + g_0(X) + U, E[U|X, D] = 0\\ D = m_0(X) ...
• 697
183 views

### Regression vs. XGBoost vs deep learning for tabular data

I am modelling a tabular dataset and getting better results with a DNN than XGB or linear regression models. The literature tells me this shouldn't be the case. Are there any specific instances when ...
1 vote
100 views

### Choosing a metric for lightGBM classifier (mean average precision k)

I have developed a binary classifier using LightGBM, where I've primarily used the AUC metric due to its simplicity, ease of use, and interpretability. Recently, I've taken an interest in utilizing ...
28 views

### Continue train xgboost specifically on misclassified observations?

I'm considering integrating the Boosting technique into a basic XGBoost classification model, in which I'd focus on misclassified instances. Assuming I have already used ...
• 121
55 views

### Using XGBoost to learn insights on data itself?

I am planning to use XGBoost to fit spatio temporal dataset that I made into a column row format. My goal isn’t prediction, but rather a regression with good fit. I just want a model fit that’s good ...
1 vote
67 views

### Poor classification even after oversampling minority class

I want to predict mortality so minority class (dead=1) is important for me but my XGBoost model performing poorly for this class. In other words, the model performed the opposite of what I wanted. the ...
119 views

### SHAP algorithm for feature selecion

I want to perform feature selection on my data. I have too many features, about 50 - 60, for not so much samples. Until today I was using the importance function of the ...
12 views

### Custom Weighs on Errors while Training

I have a linear regression model, a XGBoost model, and a MLP model that I've developed for a dataset that predicts a binary match outcome using sckit. I want to set a rule on my model where certain ...
22 views

### XGboost validation immediately drops and becomes stationary

I'm attempting to fit an xgboost model to some data. During training I'm seeing the RMSE for the validation set very quickly decrease, and then become basically static. The Validation performance is ...
43 views

### Learning Curve on two classifiers

I am using a logistic regression classifier and Gridsearch to optimize negative log loss. After generating its learning curve to test for overfitting/underfitting, my model seems to return this ...
1 vote
174 views

### XGBoost Probabilities are unrealistic

I have a model that uses XGBoost to predict a binary classification. However I am getting probability outputs for my model prediction on certain datasets that are quite unrealistic: probabilities that ...
5 views

### Mirror Entries; Benefits vs Issues

I am working on a binary predictive model using logistic regression and XGBoost and had a dataset with mirrored training sets. For example take these two dataframes ...
434 views

### How do scale_pos_weight and sample_weight interact in XGBoost?

Question: How do scale_pos_weight and sample_weight interact in XGBoost when used at the same time during training? Are they multiplied, added or something else? Example when added: Example when ...
• 443
1 vote
15 views

### XGBoost Training Logloss dropping but Validation staying steady [duplicate]

Im currently hyper parameter tuning my model and returning the model with the least amount of error. Before I start the hyper parameter tuning process I ensure my validation and test data is is ...
1 vote
40 views

### Population stability index of XGBoost model

Can a Population stability index (PSI) be calculated for the XGBoost splits? When I make a logistic regression with grouped variables, I can make PSI with the groups and see if the population in the ...
• 11
5 views

### How to represent varying reliability of ratios calculations in a dataset?

I want to predict whether the client will renew his/her subscription based on groceries consumption patterns. Suppose an order contain only one type of grocery. I have a DataFrame containing ratios of ...
52 views

### XGBoost Algorithm: Impact of Scaling Instance Weights

Does scaling the instance weights impact the XGBoost ML Model training? The instance weights determine the importance of each data point during the training process. The XGBoost ML Model uses instance ...
428 views

### Is there an implementation of xgboost for a single target variable but using multiple regression parameters

Is it possible, using custom cost functions or otherwise, to run xgboost regression for a single target variable, but rather than outputting just the conditional mean (conditioned on the feature ...
• 2,354
1 vote
42 views

### Why use a formal survival model rather than including time as a factor in my model?

My team has been implementing a survival-analysis approach to loan default prediction for about a year now. However, I believe their method to be a little "brute-force" and I haven't been ...
• 11
44 views

### NA's in Random Forest and Gradient Boosting

I have a dataset with many observations with NA's in one variable (almost a third of them). Actually, is a numerical variable with some values being zero (these zeroes mean 'no data') My options: Try ...
• 219