Skip to main content

Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Extremely high logloss in binary classification problem [duplicate]

I have a binary classification problem that I am currently trying to tackle with xgboost. This is a low signal-to-noise ratio situation dealing with time series. Per this answer "Dumb" log-...
Baron Yugovich's user avatar
0 votes
1 answer
65 views

Logloss worse than random guessing with xgboost

I have a binary classification problem that I am currently trying to tackle with xgboost. This is a low signal-to-noise ratio situation dealing with time series. My out of sample AUC is 0.65, which is ...
Baron Yugovich's user avatar
0 votes
0 answers
6 views

Using scale pos weight and non 0.5 cut off score for a look-alike model

I'm working on a classification problem where I'm trying to identify look-alikes of Class 1 in Class 0. Class 1 and Class 0 are established based on type of product customers use. Basically, Class 1 ...
user3437212's user avatar
0 votes
0 answers
6 views

Why the different default parameters for scikit-learn gradient boosting classifiers? (GradientBoostingClassifier and HistGradientBoostingClassifier)

Why do gradient boosting classifiers (GradientBoostingClassifier) and histogram-based gradient boosting classifiers (HistGradientBoostingClassifier) have significantly different default hyperparameter ...
Grendel13G's user avatar
2 votes
0 answers
28 views

SHAP values under multicolinearity/feature dependence

My task is to explain individual predictions, but having read the original paper and sifted through the internet, I am still unsure whether using something like TreeSHAP can help me with the situation ...
Saashe's user avatar
  • 21
2 votes
1 answer
25 views

Efficient prediction using Lightgbm/XGBoost when varying single feature keeping the remaining constant

Assume we have a pre-trained Lightgbm/XGBoost model $f$ dependent on the feature matrix: $$X=\left[z, C\right]$$ Here $z$ is a single feature column and $C$ is the remaining feature columns. I need to ...
BLaursen's user avatar
  • 293
0 votes
0 answers
20 views

Information coefficient as loss function of XGBoost

I am trying to train an XGBoost regressor for stock price prediction. I want to customize the objective function to be Information Coefficient (IC). The definition of IC is the Pearson correlation ...
atlantic0cean's user avatar
11 votes
2 answers
416 views

(THEORY) Do Tree models output probabilities?

I have a question purely theoretical about decision trees outputs for classification. I have heard a lot of people say "the output of tree models are not probabilities", and having studied ...
Felipe Araya Olea's user avatar
0 votes
0 answers
32 views

Neutral number in XGBoost algorithm prediction

Does a concept of "neutral number" in machine learning algorithms exist? To make it clearer: suppose we have a logistic regression with only one feature, the "neutral number" is ...
Marco Ballerini's user avatar
1 vote
0 answers
22 views

On the History of Gradient Boosting

I have recently done some work altering popular gradient boosted decision trees (GBDTs) for regression, and I was just working on establishing a theoretic basis for the modern algorithm. There is a ...
jeffery_the_wind's user avatar
0 votes
0 answers
15 views

Multiple regression model for devices located in different countries

How can I deal with a regression problem where I have a group of time-series signals (40) and predict a few features, the situation is that the data comes from different devices located around the ...
Yassin's user avatar
  • 101
0 votes
0 answers
15 views

Enforcing symmetries "for bag-of-vector" data in XGBoost or random forest - geodata example for illustration

I'll give a concrete toy problem, then give some comments on what sorts of abstractions I care about. Toy problem: Each person $i$ in my dataset has a phone, and every once in a while the phone will ...
user1557414's user avatar
2 votes
1 answer
32 views

XGBoost: does manipulating the sample make it "extrapolate"?

Suppose I want to perform time series forecasting with XGBoost. I understand that tree-based models cannot extrapolate. However, the time series I am working with is stationary (no trend or obvious ...
Mr. Ivan's user avatar
2 votes
1 answer
49 views

BerHu custom loss function for XGBoost

I would like a loss function that penalizes outliers like the squared loss, while treating small errors less sharply, like the absolute loss. It seems that I am looking for a Huber loss function, but ...
Mr. Ivan's user avatar
1 vote
0 answers
57 views

An error occurred when using the xgboost as a classifier for hiclass [closed]

Bellow it's my example when using the xgboost classifier for hiclass. My question is specifically directed to the hiClass Python package for hierarchical classification. I would like to model the ...
Ramzy's user avatar
  • 21
3 votes
0 answers
40 views

XGBoost with time lagged predictors

I have a prediction problem that involves an outcome $Y_t$ and predictors $X_t$ that vary with time $t$. I want to fit a regression of $Y_t$ on $(t,X_t)$ including also lagged versions of $X$, i.e., $...
Iván Díaz's user avatar
1 vote
0 answers
27 views

XGB predict_proba estimates don't match sum of leaves [closed]

When using an XGB model in the context of binary classification, I observed that the test estimates given by predict_proba were close but not equal to the results I ...
Juan Felipe Salamanca Lozano's user avatar
1 vote
0 answers
28 views

Is stationarity important when using boosting models?

I've studied time series for the past months and I've seen mainly two ways of building a forecasting model: Using ensemble algorithms and making the time series look like a cross-sectional data, in ...
trder's user avatar
  • 660
3 votes
1 answer
61 views

Training a model where true value is only known in batches

I would like a model to learn a function $f(x) = y$ based on some data, but suppose that I only know the true value in small batches. More precisely, each training sample consists of $x_1,\ldots,x_n$ ...
Gazerun's user avatar
  • 133
0 votes
0 answers
22 views

Missing features in decision tree based algorithms

I have a medium-sized dataset consisting of many features, some of which can contain missing values. I want to predict a variable using an algorithm that employs decision trees (specifically XGBoost, ...
umbal's user avatar
  • 75
0 votes
0 answers
8 views

Introducing bias via combining probability outputs from multiple models

I am working on a classification task, where I am trying to estimate the probability that a patient may not die. I did use a Survival Analysis approach at first, but the results seemed unintuitive and ...
vjgu's user avatar
  • 23
0 votes
0 answers
67 views

Point-level prediction intervals in LightGBM models

I would like to compute prediction intervals for LightGBM at the sample level. In other words, given a certain row to be classified (supervised classification, not regression), what is the upper bound ...
Tiago Melo's user avatar
3 votes
1 answer
57 views

Features available during training but not at prediction

Broadly, my motivation is to understand if/how features available during training but not at prediction can be used to improve the prediction accuracy of a machine learning model. This question is ...
Foster's user avatar
  • 31
0 votes
0 answers
18 views

Income Prediction using Joint Classifier and Regression Models

so I have an Income Regression task for which I'm building an XGBoost Model. When I built my first model there was an issue that the low incomes would get highly overestimated and high incomes ...
Ebrin's user avatar
  • 111
0 votes
0 answers
9 views

Weighting F1 score in a way to preference FP minimization over FN

We have a use case where we are using F1 score to optimize threshold selection for a binary classifier. In this use case however, FP's present a higher risk impact than FN. We'd like to consider a way ...
F1_score_tuned_for_usecase's user avatar
1 vote
1 answer
60 views

How does Cross Validation work in decision trees (or tree ensembles)

I've been working with tree-based models for a long time and I never really asked myself how cross-validation would work when building a tree. For the sake of this question, suppose I've split my ...
Arturo Sbr's user avatar
2 votes
1 answer
61 views

Tuning the learning rate parameter in GBDT models

I've always been taught that decreasing the learning rate parameter in gbdt models such as XGBoost, LightGBM and Catboost will improve the out-of-sample performance, assuming the number of iterations ...
Casper's user avatar
  • 21
2 votes
1 answer
54 views

XGBoost original paper equation simplification

In the original XGboost paper (https://arxiv.org/abs/1603.02754), in section 3.3, the authors simplified equation 3: To: Shouldn't this be a positive sign with the target being -g/h? Wikipedia shows ...
Z Li's user avatar
  • 123
1 vote
1 answer
45 views

What is the intuition for estimating residuals when boosting linear regression models?

So basically the title is my question. lin-reg model: $$y_i = x^{T}_i\beta + \epsilon_i, i = 1,...,n$$ Initalize $\hat{\beta^{[0]}}$ and the number of iterations $m_{stop}$. Compute: $$u = y - X\hat{\...
BlankerHans's user avatar
0 votes
0 answers
39 views

How to diagnose the number of bootstraps I need?

I am running an XGBoost model to predict the global economic cost of invasive species. My training set is only about 3000 data points. I am bootstrapping my predictions, and went with the default 1000 ...
Gabriel De Oliveira Caetano's user avatar
0 votes
0 answers
56 views

What are the benefits of using pseudo-residuals in Gradient Boosting?

At each iteration $t$ of the Gradient Boosting algorithm, we're basically trying to add the weak learner $f_t$ that minimizes: $$ \mathcal{L}_t = \sum\limits_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(\...
Druudik's user avatar
  • 143
1 vote
0 answers
91 views

Best practice for subsampling training data and weights (in XGBoost)

I am trying to build an XGBoost model in pycharm and I have a general method question even though it relates to my model of choice (XGBoost). Any kind of general comments on the proper statistical ...
Magi's user avatar
  • 41
0 votes
0 answers
28 views

Hyperdimensional computing versus gradient boosting and NN on tabular data

I've been trying to learn hyperdimensional computing (aka vector symbolic architectures). There's not a lot of resources out there. I've found a few examples, but I can't seem to get very good results ...
Dan's user avatar
  • 111
0 votes
0 answers
27 views

Am I able to calculate SHAP directly for my testing dataset?

I trained an XGBoost classifier model on a training set, and I predicted it on the testing set. I also calculated the respective class prediction values. I concatenated X_test and y_test together. I ...
bob jones's user avatar
3 votes
0 answers
97 views

Gradient boosting and quantile regression performance issues [closed]

My goal is to develop a ML model that predicts the remaining flight time. To do so I have different features: distance altitude speed vertical rate Here is a plot showing the actual remaining time ...
bfgt's user avatar
  • 317
0 votes
0 answers
12 views

Linear SVM vs Decision Stumps for AdaBoost

I have heard that AdaBoost can use a linear SVM as a weak classificer. I wonder why Decision Stumps is often used with AdaBoost? Booth are binary classifiers. In my opinion, linear SVM seems to be a ...
euraad's user avatar
  • 425
0 votes
0 answers
10 views

What's the purpose by returning back the bad outliers from AdaBoost?

Assume that you having a matrix $X$ that holds both numerical and binary data. You plug in the data $X$ into AdaBoost and AdaBoost update the $X$ by only focusing on the rows that could not make a ...
euraad's user avatar
  • 425
4 votes
1 answer
69 views

Overfitting GBM by simultaneously adding trees and lowering learning rate?

I understand that you can overfit a Gradient Boosting Machine (GBM) by using too many trees (unlike random forest), and also that you can overfit a GBM by using too high of a learning rate. My ...
David's user avatar
  • 1,276
6 votes
1 answer
150 views

XGBoost Learning to Rank with XGBClassifier

I am trying to build a model trained on binary labels that has a high precision for the top k predicted instances, and don’t care too much about recall or precision more generally. I was then ...
A. Bollans's user avatar
5 votes
1 answer
121 views

approches for linear extrapolation of xgboost model tails

I would like some insight on known approaches for linear extrapolation on tails of xgboost models. The current model is missing data at the distribution tails and is thus predicting flat trends for ...
aort01's user avatar
  • 151
0 votes
1 answer
91 views

Is there a way to enforce factor importance in random forest/xgboost

Suppose I have 3 predictors to predict stock returns. 1 of the 3 is known for ages and is still doing well. The rest 2 are newly found ones. So in a crude portfolio construction fashion, I'd do $s = 0....
jf328's user avatar
  • 811
1 vote
0 answers
28 views

Partial derivative notation extreme gradient boosting

What is meant by this notation regarding gradient boosting source: $$ g_i = \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)}) $$ Is it the partial derivative of $l$ w.r.t. $\hat{y}_i^{(t-1)}$ , i....
Herbert's user avatar
  • 165
2 votes
1 answer
174 views

How to interpret the deviance plot by boosting models

This plot is taken from a gradient boosting regression example in the scikit-learn documentation. What does deviance mean? How should this plot be interpreted? In which case do we have over/...
Mykola Zotko's user avatar
0 votes
0 answers
37 views

XGBoost's subsample = 0?

In my use of XGBoost with the gradient-based method, I inadvertently set subsample to 0, yet it surprisingly returned a good result. I am not sure how to explain it well. Any idea from the community?
Mel Huang's user avatar
1 vote
1 answer
154 views

Help with Classification model for S&P500

I have started a project in order to develop my coding skills, where I am predicting next month's S&P500 return direction based on some macroeconomic and financial variables. These datasets have ...
user199's user avatar
  • 23
3 votes
1 answer
267 views

Ordinal vs multinominal classification in XGboost: differences in one-hot encoding

I have followed this post and tried to see if there will be any difference in predicted probabilities if I use different one-hot encoding in XGboost. This is my code with some dummy data, which is ...
deblue's user avatar
  • 243
7 votes
1 answer
514 views

Ordinal log-loss in a multiclass classification in XGBoost?

I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High). What I would ...
deblue's user avatar
  • 243
1 vote
1 answer
135 views

Multivariate Time Series dataset preparation

I am a bit confused with the time series dataset preparation. From the internet, I saw all examples which used tree-based models, had input features and target defined as: ...
kg__'s user avatar
  • 63
0 votes
0 answers
75 views

Cross-validation and automated binning of a continuous variable for a continuous target

I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
hexolitemax's user avatar
0 votes
0 answers
47 views

Best practices when training an xgboost model as part of a larger model?

I would like to train a model end-to-end that uses the output from an xgboost model as an input. I've successfully implemented full-batch gradient descent into my pipeline with jax, following this ...
ironicoxidant's user avatar

1
2 3 4 5
30