Questions tagged [boosting]
A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.
1,445
questions
1
vote
0
answers
23
views
Partial derivative notation extreme gradient boosting
What is meant by this notation regarding gradient boosting source:
$$
g_i = \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)})
$$
Is it the partial derivative of $l$ w.r.t. $\hat{y}_i^{(t-1)}$ , i....
- 165
2
votes
1
answer
94
views
How to interpret the deviance plot by boosting models
This plot is taken from a gradient boosting regression example in the scikit-learn documentation.
What does deviance mean? How should this plot be interpreted? In which case do we have over/...
- 55
0
votes
0
answers
13
views
XGBoost's subsample = 0?
In my use of XGBoost with the gradient-based method, I inadvertently set subsample to 0, yet it surprisingly returned a good result.
I am not sure how to explain it well. Any idea from the community?
1
vote
1
answer
117
views
Help with Classification model for S&P500
I have started a project in order to develop my coding skills, where I am predicting next month's S&P500 return direction based on some macroeconomic and financial variables. These datasets have ...
- 13
2
votes
1
answer
33
views
Ordinal vs multinominal classification in XGboost: differences in one-hot encoding
I have followed this post and tried to see if there will be any difference in predicted probabilities if I use different one-hot encoding in XGboost.
This is my code with some dummy data, which is ...
- 221
5
votes
1
answer
112
views
Ordinal log-loss in a multiclass classification in XGBoost?
I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High).
What I would ...
- 221
1
vote
1
answer
94
views
Multivariate Time Series dataset preparation
I am a bit confused with the time series dataset preparation. From the internet, I saw all examples which used tree-based models, had input features and target defined as:
...
- 61
0
votes
0
answers
34
views
Cross-validation and automated binning of a continuous variable for a continuous target
I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
0
votes
0
answers
20
views
Best practices when training an xgboost model as part of a larger model?
I would like to train a model end-to-end that uses the output from an xgboost model as an input. I've successfully implemented full-batch gradient descent into my pipeline with jax, following this ...
1
vote
0
answers
38
views
Assigning class weights during XGBoost training phase [closed]
I am using the following code to train a XGBoost model. I have specified the weights in random_search fit. My question is, does this adjust the weights during training fit or should I include "...
- 11
1
vote
1
answer
75
views
How to use priors to impute values at an individual level and replicate a distribution of the population?
I am trying to correct a variable from a survey that has measurement error. To do this, I have been taking this column as if it was missing and imputing new values based on the predictions of an ...
1
vote
1
answer
64
views
XGBoost - Linear Tree
I’ve been reading about linear tree models particularly the linear-tree package and the option to use linear trees in LightGBM if one sets the parameter linear_tree ...
- 11
0
votes
0
answers
30
views
Why are SHAP interaction values inconsistent with those from a linear model?
I have fit a gradient boosted regressor to some tabular data.
When looking at dependence plots there are some clear interaction effects e.g. effect of gender on the outcome measure reverses dependent ...
- 233
1
vote
0
answers
9
views
Avoiding over-fitting in Gradient Boosted tree models when multiple sequential observations share the same label [closed]
I am trying to train a Multi-class classification model where every K minutes I receive a set of features x and use the set to ...
2
votes
0
answers
48
views
XGBoost Calibration for weighted loss function
I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
- 45
0
votes
0
answers
20
views
random forest/ boosting and overfitting issue
I guess we can use CV to optimize how many trees (i.e. ntree or nround for boosting) in order to not overfit. It's like one of the way to not overfit.
If we have too many complex trees that we learned ...
- 53
0
votes
0
answers
20
views
Why my XGBoostClassifier model results in perfect accuracy despite dropping corelated features?
I am trying to do a binary classification on ticket canceling data from kaggle.
I know this question has been asked before. For example here and here
Summary of what I learned in those references:
...
- 101
0
votes
0
answers
46
views
XGBoost imbalanced class problem
I am currently working on a personal project for predicting financial equities movements into 3 classes. The three classes are 1.) above a certain percentage move within a certain timeframe, 2.) below ...
0
votes
0
answers
16
views
Update/ recalibrate XG Boost, Random Forest, GLM models for external validation
I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects.
I have ...
- 11
0
votes
0
answers
89
views
Monotone constraints in decision tree regressor or random forest regression
after I've spent several weeks trying to fit a regression model to my flood damage data (x1=water height, x2=adaptation height, x3=(x1-x2), y=damage), it is now time for my very first question on ...
- 11
3
votes
1
answer
64
views
Can XGBoost learn more complicated interactions/features?
For a set of features {a, b, c, d . . . n}, XGBoost can easily learn, say, a*d. In practice can it also effectively learn a/c? Or (a + b + c + 2)/d? Or (c^(2d))/(b^a)? I'd imagine some of this depends ...
- 161
1
vote
1
answer
52
views
Correct probability with XGBoost in R
It seems that I don't understand results of XGBoost. I've made a model with one variable and
a binary target variable, and only one tree.
Sample of data structure is presented below.
...
- 13
1
vote
0
answers
58
views
Understanding a calibration plot for lightGBM binary classifier
I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
- 489
0
votes
1
answer
65
views
Trouble Achieving Optimal Parameters for XGBoost Regressor with Sin(x) Time Series
I'm facing a challenge while attempting to optimize parameters for an XGBoost regressor (Python) using time series data. I've created a time series using the following code:
...
2
votes
0
answers
26
views
What algorithm/approach is best for a multiclass classifier where there are a significant amount of misclassifications in the source data?
I am using document embeddings (100 dimensions) to train a classifier on text data, however, I am getting poor results which I am attributing to the fact that there are a large proportion of ...
- 31
1
vote
0
answers
63
views
Unusual results from XGBoost learning curves
I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
- 21
0
votes
0
answers
37
views
Can I add the gain feature importance metric for multiple features from XGBoost?
I would like to combine the "gain" feature importance metric from multiple features since this is potentially more informative. In my example, I would like to compare the importance of both ...
0
votes
1
answer
36
views
Why is the initial prediction of gradient boosting classifier based on log of odds? Why can't we work with probabilities instead?
I am unable to understand the role of log odds in GBM classification. Why can't we just work with probabilities?
0
votes
1
answer
16
views
Working with subsets of values from single category in XGBoost
Since version 1.5, XGBoost supports categorical data out of the box, which is a convenient way to skip the one-hot pre-processing step and allow for if X in values ...
- 262
4
votes
1
answer
57
views
Double Machine Learning: What kind of "naive" estimator do the authors use to get such a bias?
I am reading the double-machine learning paper by Chernozhukov et al. (2018), in Example 1.1. the authors consider the partially linear model:
$$Y = D \theta_0 + g_0(X) + U, E[U|X, D] = 0\\ D = m_0(X) ...
- 697
0
votes
0
answers
183
views
Regression vs. XGBoost vs deep learning for tabular data
I am modelling a tabular dataset and getting better results with a DNN than XGB or linear regression models. The literature tells me this shouldn't be the case. Are there any specific instances when ...
1
vote
0
answers
100
views
Choosing a metric for lightGBM classifier (mean average precision k)
I have developed a binary classifier using LightGBM, where I've primarily used the AUC metric due to its simplicity, ease of use, and interpretability.
Recently, I've taken an interest in utilizing ...
- 489
2
votes
0
answers
28
views
Continue train xgboost specifically on misclassified observations?
I'm considering integrating the Boosting technique into a basic XGBoost classification model, in which I'd focus on misclassified instances. Assuming I have already used ...
- 121
2
votes
1
answer
55
views
Using XGBoost to learn insights on data itself?
I am planning to use XGBoost to fit spatio temporal dataset that I made into a column row format. My goal isn’t prediction, but rather a regression with good fit.
I just want a model fit that’s good ...
1
vote
1
answer
67
views
Poor classification even after oversampling minority class
I want to predict mortality so minority class (dead=1) is important for me but my XGBoost model performing poorly for this class. In other words, the model performed the opposite of what I wanted.
the ...
- 23
3
votes
1
answer
119
views
SHAP algorithm for feature selecion
I want to perform feature selection on my data. I have too many features, about 50 - 60, for not so much samples.
Until today I was using the importance function of the ...
- 489
0
votes
0
answers
12
views
Custom Weighs on Errors while Training
I have a linear regression model, a XGBoost model, and a MLP model that I've developed for a dataset that predicts a binary match outcome using sckit.
I want to set a rule on my model where certain ...
- 25
0
votes
0
answers
22
views
XGboost validation immediately drops and becomes stationary
I'm attempting to fit an xgboost model to some data. During training I'm seeing the RMSE for the validation set very quickly decrease, and then become basically static. The Validation performance is ...
- 1
0
votes
0
answers
43
views
Learning Curve on two classifiers
I am using a logistic regression classifier and Gridsearch to optimize negative log loss.
After generating its learning curve to test for overfitting/underfitting, my model seems to return this ...
- 25
1
vote
1
answer
174
views
XGBoost Probabilities are unrealistic
I have a model that uses XGBoost to predict a binary classification. However I am getting probability outputs for my model prediction on certain datasets that are quite unrealistic: probabilities that ...
- 25
0
votes
0
answers
5
views
Mirror Entries; Benefits vs Issues
I am working on a binary predictive model using logistic regression and XGBoost and had a dataset with mirrored training sets. For example take these two dataframes
...
- 25
0
votes
1
answer
434
views
How do scale_pos_weight and sample_weight interact in XGBoost?
Question:
How do scale_pos_weight and sample_weight interact in XGBoost when used at the same time during training? Are they multiplied, added or something else?
Example when added:
Example when ...
- 443
1
vote
0
answers
15
views
XGBoost Training Logloss dropping but Validation staying steady [duplicate]
Im currently hyper parameter tuning my model and returning the model with the least amount of error. Before I start the hyper parameter tuning process I ensure my validation and test data is is ...
- 11
1
vote
0
answers
40
views
Population stability index of XGBoost model
Can a Population stability index (PSI) be calculated for the XGBoost splits?
When I make a logistic regression with grouped variables, I can make PSI with the groups and see if the population in the ...
- 11
0
votes
0
answers
5
views
How to represent varying reliability of ratios calculations in a dataset?
I want to predict whether the client will renew his/her subscription based on groceries consumption patterns. Suppose an order contain only one type of grocery.
I have a DataFrame containing ratios of ...
0
votes
0
answers
52
views
XGBoost Algorithm: Impact of Scaling Instance Weights
Does scaling the instance weights impact the XGBoost ML Model training?
The instance weights determine the importance of each data point during the training process. The XGBoost ML Model uses instance ...
- 1
3
votes
2
answers
428
views
Is there an implementation of xgboost for a single target variable but using multiple regression parameters
Is it possible, using custom cost functions or otherwise, to run xgboost regression for a single target variable, but rather than outputting just the conditional mean (conditioned on the feature ...
- 2,354
1
vote
1
answer
42
views
Why use a formal survival model rather than including time as a factor in my model?
My team has been implementing a survival-analysis approach to loan default prediction for about a year now. However, I believe their method to be a little "brute-force" and I haven't been ...
- 11
0
votes
0
answers
44
views
NA's in Random Forest and Gradient Boosting
I have a dataset with many observations with NA's in one variable (almost a third of them). Actually, is a numerical variable with some values being zero (these zeroes mean 'no data')
My options:
Try ...
- 219
0
votes
0
answers
50
views
XGBoost custom objective/loss: when is a Reverse (inverse) Link Function required?
I'm trying to implement a custom objective function in XGBoost. I read the docs on this topic. I am not sure if I need to define a "reverse link function" (aka inverse link function) to ...
- 187