Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0 votes
1 answer
40 views

What is the intuition for estimating residuals when boosting linear regression models?

So basically the title is my question. lin-reg model: $$y_i = x^{T}_i\beta + \epsilon_i, i = 1,...,n$$ Initalize $\hat{\beta^{[0]}}$ and the number of iterations $m_{stop}$. Compute: $$u = y - X\hat{\...
BlankerHans's user avatar
0 votes
0 answers
35 views

How to diagnose the number of bootstraps I need?

I am running an XGBoost model to predict the global economic cost of invasive species. My training set is only about 3000 data points. I am bootstrapping my predictions, and went with the default 1000 ...
Gabriel De Oliveira Caetano's user avatar
0 votes
0 answers
27 views

What are the benefits of using pseudo-residuals in Gradient Boosting?

At each iteration $t$ of the Gradient Boosting algorithm, we're basically trying to add the weak learner $f_t$ that minimizes: $$ \mathcal{L}_t = \sum\limits_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(\...
Druudik's user avatar
  • 143
1 vote
0 answers
39 views

Best practice for subsampling training data and weights (in XGBoost)

I am trying to build an XGBoost model in pycharm and I have a general method question even though it relates to my model of choice (XGBoost). Any kind of general comments on the proper statistical ...
Magi's user avatar
  • 41
0 votes
0 answers
20 views

Hyperdimensional computing versus gradient boosting and NN on tabular data

I've been trying to learn hyperdimensional computing (aka vector symbolic architectures). There's not a lot of resources out there. I've found a few examples, but I can't seem to get very good results ...
Dan's user avatar
  • 111
0 votes
0 answers
18 views

Am I able to calculate SHAP directly for my testing dataset?

I trained an XGBoost classifier model on a training set, and I predicted it on the testing set. I also calculated the respective class prediction values. I concatenated X_test and y_test together. I ...
bob jones's user avatar
3 votes
0 answers
60 views

Gradient boosting and quantile regression performance issues [closed]

My goal is to develop a ML model that predicts the remaining flight time. To do so I have different features: distance altitude speed vertical rate Here is a plot showing the actual remaining time ...
bfgt's user avatar
  • 307
0 votes
0 answers
11 views

Linear SVM vs Decision Stumps for AdaBoost

I have heard that AdaBoost can use a linear SVM as a weak classificer. I wonder why Decision Stumps is often used with AdaBoost? Booth are binary classifiers. In my opinion, linear SVM seems to be a ...
euraad's user avatar
  • 415
0 votes
0 answers
10 views

What's the purpose by returning back the bad outliers from AdaBoost?

Assume that you having a matrix $X$ that holds both numerical and binary data. You plug in the data $X$ into AdaBoost and AdaBoost update the $X$ by only focusing on the rows that could not make a ...
euraad's user avatar
  • 415
2 votes
1 answer
33 views

Overfitting GBM by simultaneously adding trees and lowering learning rate?

I understand that you can overfit a Gradient Boosting Machine (GBM) by using too many trees (unlike random forest), and also that you can overfit a GBM by using too high of a learning rate. My ...
DavidSilverberg's user avatar
6 votes
1 answer
50 views

XGBoost Learning to Rank with XGBClassifier

I am trying to build a model trained on binary labels that has a high precision for the top k predicted instances, and don’t care too much about recall or precision more generally. I was then ...
A. Bollans's user avatar
4 votes
1 answer
51 views

approches for linear extrapolation of xgboost model tails

I would like some insight on known approaches for linear extrapolation on tails of xgboost models. The current model is missing data at the distribution tails and is thus predicting flat trends for ...
aort01's user avatar
  • 141
0 votes
1 answer
56 views

Is there a way to enforce factor importance in random forest/xgboost

Suppose I have 3 predictors to predict stock returns. 1 of the 3 is known for ages and is still doing well. The rest 2 are newly found ones. So in a crude portfolio construction fashion, I'd do $s = 0....
jf328's user avatar
  • 811
1 vote
0 answers
27 views

Partial derivative notation extreme gradient boosting

What is meant by this notation regarding gradient boosting source: $$ g_i = \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)}) $$ Is it the partial derivative of $l$ w.r.t. $\hat{y}_i^{(t-1)}$ , i....
Herbert's user avatar
  • 165
2 votes
1 answer
116 views

How to interpret the deviance plot by boosting models

This plot is taken from a gradient boosting regression example in the scikit-learn documentation. What does deviance mean? How should this plot be interpreted? In which case do we have over/...
Mykola Zotko's user avatar
0 votes
0 answers
20 views

XGBoost's subsample = 0?

In my use of XGBoost with the gradient-based method, I inadvertently set subsample to 0, yet it surprisingly returned a good result. I am not sure how to explain it well. Any idea from the community?
Mel Huang's user avatar
1 vote
1 answer
131 views

Help with Classification model for S&P500

I have started a project in order to develop my coding skills, where I am predicting next month's S&P500 return direction based on some macroeconomic and financial variables. These datasets have ...
user199's user avatar
  • 13
3 votes
1 answer
76 views

Ordinal vs multinominal classification in XGboost: differences in one-hot encoding

I have followed this post and tried to see if there will be any difference in predicted probabilities if I use different one-hot encoding in XGboost. This is my code with some dummy data, which is ...
deblue's user avatar
  • 231
5 votes
1 answer
239 views

Ordinal log-loss in a multiclass classification in XGBoost?

I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High). What I would ...
deblue's user avatar
  • 231
1 vote
1 answer
103 views

Multivariate Time Series dataset preparation

I am a bit confused with the time series dataset preparation. From the internet, I saw all examples which used tree-based models, had input features and target defined as: ...
kg__'s user avatar
  • 63
0 votes
0 answers
50 views

Cross-validation and automated binning of a continuous variable for a continuous target

I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
hexolitemax's user avatar
0 votes
0 answers
29 views

Best practices when training an xgboost model as part of a larger model?

I would like to train a model end-to-end that uses the output from an xgboost model as an input. I've successfully implemented full-batch gradient descent into my pipeline with jax, following this ...
ironicoxidant's user avatar
1 vote
1 answer
86 views

How to use priors to impute values at an individual level and replicate a distribution of the population?

I am trying to correct a variable from a survey that has measurement error. To do this, I have been taking this column as if it was missing and imputing new values based on the predictions of an ...
Santiago Valdivieso's user avatar
1 vote
1 answer
215 views

XGBoost - Linear Tree

I’ve been reading about linear tree models particularly the linear-tree package and the option to use linear trees in LightGBM if one sets the parameter linear_tree ...
harrynak's user avatar
0 votes
0 answers
54 views

Why are SHAP interaction values inconsistent with those from a linear model?

I have fit a gradient boosted regressor to some tabular data. When looking at dependence plots there are some clear interaction effects e.g. effect of gender on the outcome measure reverses dependent ...
RobMcC's user avatar
  • 233
1 vote
0 answers
10 views

Avoiding over-fitting in Gradient Boosted tree models when multiple sequential observations share the same label [closed]

I am trying to train a Multi-class classification model where every K minutes I receive a set of features x and use the set to ...
Dean Grosbard's user avatar
2 votes
1 answer
110 views

XGBoost Calibration for weighted loss function

I am currently using XGBoost (in R) to perform multiclass classification. I am using merror=eval_metric and my objective is <...
HeyCool08's user avatar
0 votes
0 answers
23 views

random forest/ boosting and overfitting issue

I guess we can use CV to optimize how many trees (i.e. ntree or nround for boosting) in order to not overfit. It's like one of the way to not overfit. If we have too many complex trees that we learned ...
user398060's user avatar
0 votes
0 answers
22 views

Why my XGBoostClassifier model results in perfect accuracy despite dropping corelated features?

I am trying to do a binary classification on ticket canceling data from kaggle. I know this question has been asked before. For example here and here Summary of what I learned in those references: ...
wander95's user avatar
  • 101
0 votes
0 answers
46 views

XGBoost imbalanced class problem

I am currently working on a personal project for predicting financial equities movements into 3 classes. The three classes are 1.) above a certain percentage move within a certain timeframe, 2.) below ...
Victor Minin's user avatar
0 votes
0 answers
25 views

Update/ recalibrate XG Boost, Random Forest, GLM models for external validation

I have created XG Boost, Random Forest and GLM models for classification of a binary outcome and now I want to externally validate the models on a different population of over 5000 subjects. I have ...
mmo's user avatar
  • 11
0 votes
0 answers
182 views

Monotone constraints in decision tree regressor or random forest regression

after I've spent several weeks trying to fit a regression model to my flood damage data (x1=water height, x2=adaptation height, x3=(x1-x2), y=damage), it is now time for my very first question on ...
Sjafnargata's user avatar
4 votes
1 answer
82 views

Can XGBoost learn more complicated interactions/features?

For a set of features {a, b, c, d . . . n}, XGBoost can easily learn, say, a*d. In practice can it also effectively learn a/c? Or (a + b + c + 2)/d? Or (c^(2d))/(b^a)? I'd imagine some of this depends ...
BigMistake's user avatar
1 vote
1 answer
62 views

Correct probability with XGBoost in R

It seems that I don't understand results of XGBoost. I've made a model with one variable and a binary target variable, and only one tree. Sample of data structure is presented below. ...
Michal's user avatar
  • 13
5 votes
0 answers
83 views

Understanding a calibration plot for lightGBM binary classifier

I wanted to assess the performance of my lightGBM classifier using a calibration plot. If I understood correctly, a calibration plot visualizes the alignment between the predicted probabilities by the ...
Programming Noob's user avatar
0 votes
1 answer
86 views

Trouble Achieving Optimal Parameters for XGBoost Regressor with Sin(x) Time Series

I'm facing a challenge while attempting to optimize parameters for an XGBoost regressor (Python) using time series data. I've created a time series using the following code: ...
Zoi Lisoi's user avatar
2 votes
0 answers
28 views

What algorithm/approach is best for a multiclass classifier where there are a significant amount of misclassifications in the source data?

I am using document embeddings (100 dimensions) to train a classifier on text data, however, I am getting poor results which I am attributing to the fact that there are a large proportion of ...
osckt's user avatar
  • 31
1 vote
0 answers
81 views

Unusual results from XGBoost learning curves

I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
Ted's user avatar
  • 21
0 votes
0 answers
64 views

Can I add the gain feature importance metric for multiple features from XGBoost?

I would like to combine the "gain" feature importance metric from multiple features since this is potentially more informative. In my example, I would like to compare the importance of both ...
Felix Richter's user avatar
0 votes
1 answer
88 views

Why is the initial prediction of gradient boosting classifier based on log of odds? Why can't we work with probabilities instead?

I am unable to understand the role of log odds in GBM classification. Why can't we just work with probabilities?
user394273's user avatar
0 votes
1 answer
23 views

Working with subsets of values from single category in XGBoost

Since version 1.5, XGBoost supports categorical data out of the box, which is a convenient way to skip the one-hot pre-processing step and allow for if X in values ...
Alexandru Dinu's user avatar
4 votes
1 answer
60 views

Double Machine Learning: What kind of "naive" estimator do the authors use to get such a bias?

I am reading the double-machine learning paper by Chernozhukov et al. (2018), in Example 1.1. the authors consider the partially linear model: $$Y = D \theta_0 + g_0(X) + U, E[U|X, D] = 0\\ D = m_0(X) ...
D F's user avatar
  • 707
0 votes
0 answers
236 views

Regression vs. XGBoost vs deep learning for tabular data

I am modelling a tabular dataset and getting better results with a DNN than XGB or linear regression models. The literature tells me this shouldn't be the case. Are there any specific instances when ...
EuginePickett's user avatar
1 vote
0 answers
124 views

Choosing a metric for lightGBM classifier (mean average precision k)

I have developed a binary classifier using LightGBM, where I've primarily used the AUC metric due to its simplicity, ease of use, and interpretability. Recently, I've taken an interest in utilizing ...
Programming Noob's user avatar
2 votes
0 answers
31 views

Continue train xgboost specifically on misclassified observations?

I'm considering integrating the Boosting technique into a basic XGBoost classification model, in which I'd focus on misclassified instances. Assuming I have already used ...
helloworld's user avatar
2 votes
1 answer
67 views

Using XGBoost to learn insights on data itself?

I am planning to use XGBoost to fit spatio temporal dataset that I made into a column row format. My goal isn’t prediction, but rather a regression with good fit. I just want a model fit that’s good ...
cwanderroycbooks's user avatar
1 vote
1 answer
77 views

Poor classification even after oversampling minority class

I want to predict mortality so minority class (dead=1) is important for me but my XGBoost model performing poorly for this class. In other words, the model performed the opposite of what I wanted. the ...
Nima Yousefi's user avatar
3 votes
1 answer
195 views

SHAP algorithm for feature selecion

I want to perform feature selection on my data. I have too many features, about 50 - 60, for not so much samples. Until today I was using the importance function of the ...
Programming Noob's user avatar
0 votes
0 answers
13 views

Custom Weighs on Errors while Training

I have a linear regression model, a XGBoost model, and a MLP model that I've developed for a dataset that predicts a binary match outcome using sckit. I want to set a rule on my model where certain ...
user54565's user avatar
0 votes
0 answers
32 views

XGboost validation immediately drops and becomes stationary

I'm attempting to fit an xgboost model to some data. During training I'm seeing the RMSE for the validation set very quickly decrease, and then become basically static. The Validation performance is ...
ewhelan's user avatar

1
2 3 4 5
30