Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

(XGBOOST) 5-fold cross validation test aucpr is always lower than train aucpr, is that overfitting?

I am using XGBOOST to construct a prediction model, but no matter what I do (including set gamma, subsample, eta), I will get results similar to the following picture. I did see all train and test ...
user avatar
0 votes
0 answers
13 views

which dataset to send as eval set in xgboost, catboost, and etc, when using optuna

In some boost models there are option to send eval set while fitting the model. for example: ...
user avatar
  • 67
1 vote
1 answer
15 views

Training discrete time hazard model with xgboost on right censored loan data

I am currently developing a loan default risk model using a discrete time hazard approach with xgboost. The goal is to generate a series of predicted monthly default probabilities using a new ...
user avatar
0 votes
0 answers
14 views

eval_set porpuse in XGBOOST, CATBOOST and etc [closed]

In many Python boost packages, while fyou can also send eval set itting the model on train data, for example: ...
user avatar
  • 67
0 votes
0 answers
13 views

Problem with Duplicate Data in Case of CatBoost

We know that while converting categorical variables to numerics, CatBoost uses the following formula (source: documentation): Now, suppose there are 2 duplicate data entries. Ideally, the value of $...
user avatar
1 vote
0 answers
30 views

Reducing bias when forecasting retail sales with boosting model

I'm forecasting future sales for products in retail stores, using a LightGBM model. My model has a decent forecast accuracy, but the forecasts are biased (the average forecast error is negative, the ...
user avatar
  • 11
8 votes
1 answer
426 views

Why symmetric trees are not used in xgboost

I was reading about catboost and found out that catboost uses symmetric trees. The reasons for using symmetric trees is: The choice of oblivious trees has several advantages compared to the classic ...
user avatar
1 vote
0 answers
17 views

Why isn't RandomSearchCV returning the optimum parameters for the XGBoost Model, and how can I avoid Overfitting?

I have a dataset for energy consumer customers and binary target variables with which I want to predict the churn for the customers. Counts of target values Not Churn 0: 14153 Churn 1: 1520 I have ...
user avatar
  • 31
0 votes
0 answers
6 views

How can I measure the importance of a leaf in LightGBM?

I want to understand, which leaves in my lightgbm model have low importance (If I deleted the leaf, the model wouldn't become "worse"). Which approaches exists for it? Thank you in advance!
user avatar
1 vote
1 answer
18 views

Learning Rate in Gradient Boost

How do we decide the learning rate in gradient boost? I can think of cross validation as one of the methods, but are there any other methods? Also, my teacher said in the class that 0.1 is a good ...
user avatar
0 votes
0 answers
19 views

How important is feature selection for Light Gradient Boosting Machine?

My understanding is that LGBM will, more or less, figure out on its own which features are important and which not Does it still make any sense to do any feature selection before training a LGBM model?...
user avatar
1 vote
1 answer
22 views

XGBoost Regression on a normal distribution variable produces a one sided distribution (only positive values)

I'm running a scikit-learn XGBoostRegressor with an RMSE loss function, on a variable with a distribution that is close to symmetric around 0 (think normal distribution, with a positive mean that ...
user avatar
  • 131
1 vote
0 answers
11 views

Regression Modelling with Sets of Vectors

Hi I've a regression Machine Learning problem of the following form: Input: $\{(o_i,v_i)| o_i \in R^d, v_i \in R , i \in \{1,2,3,...M\} \}$ Output: $s \in R$ I can use a neural model (probably deep ...
user avatar
  • 121
1 vote
0 answers
19 views

Does XGBoost use gradient descent? [duplicate]

XGBoost is said to be based on "gradient tree boosting" in the original paper. Reading the paper and the introduction on the official website, it seems to me that the algorithm does not use ...
user avatar
2 votes
0 answers
23 views

rule-based model out of a binary features dataset

I have the following problem. I've been given a dataset where the features are binary which can be interpret as seen or not seen. The target value is categorical. My goal is to generate a list of ...
user avatar
2 votes
1 answer
87 views

how does `subsample` parameter work in boosting algorithms like xgboost and lightgbm?

From what I know, both of them are sequential learners and only the 1st tree in the sequence gets built on the data and all the following trees that get built are to correct the mistakes of previous ...
user avatar
1 vote
1 answer
29 views

Single Decision Tree output from a XGBoost Model

Would it be methodologically acceptable to export all the different decision trees constructed by an XGBoost model and test them singularly as a potential risk classification system?
user avatar
  • 21
1 vote
1 answer
29 views

How XGBoost chooses between two features that gives the same information?

I have 10 variables in a dataset (X1, X2, .., X10) plus the binary target variable (...
user avatar
  • 243
0 votes
0 answers
13 views

Correlation between variables from a XGBoost lens?

I have 10 variables in a dataset (X1, X2, .., X10) plus the binary target variable (...
user avatar
  • 243
0 votes
0 answers
9 views

How to know if two variables with similar feature importance are replaceable in XGBoost?

I have 10 variables in a dataset (X1, X2, .., X10) plus the binary target variable (...
user avatar
  • 243
0 votes
0 answers
23 views

Why do permutation importances for my model all zeros?

I really am at a loss here. I have several tree-based binary classification models trained on a balanced dataset of ~300 samples and ~15000 features. The models have AUROC around 0.8. I want to ...
user avatar
  • 81
0 votes
1 answer
22 views

Lightgbm, time-series and spikes repeated on a yearly basis

I have a data set (time-series) with the shape {$2190$x$63$}. There are 63 variables, 2 products ($A$ and $B$) worth of 3 years of daily data, thus I have $1095$ observations per product and total of $...
user avatar
  • 87
0 votes
0 answers
28 views

Multiclass classification on top of given probability distribution from previous model

I have a multiclass classification problem that has multiple steps. Firstly, I am given a probability distribution over the classes by a base model for each sample. The task is to create a new model ...
user avatar
0 votes
0 answers
15 views

Assessing importance of interactions between categorical features

The issues with using feature_importance of models such as XGBoost, or even using packages like SHAP or ELI5, is that the results are displayed in a way that doesn'...
user avatar
  • 529
2 votes
1 answer
38 views

XGBoost interpretation of the plot in R

I am applying XGBoost implementation in R on the data with 9 columns. After training the model, I tried to plot the "multiple-in-one" tree using the xgb.plot.multi.trees() function with the ...
user avatar
  • 21
0 votes
0 answers
38 views

How is the feature interaction different in XGBoost vs. a fully connected neural network

When you know your features could interact with each other, will you choose XGBoost or NN-based models? My friend is training with an XGBoost, and he manually adds interacted features (X1 * X2) as new ...
user avatar
  • 31
1 vote
1 answer
17 views

Random Forests- Out of Bag Error Calculation

I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
user avatar
0 votes
0 answers
12 views

Are tree based algorithms weak at correctly predicting an outcome for a given variable range, if there is sparse data for that variable range?

I am using XGBoost as a classifier, and one of my important variables has values ranging from 500 to 20000. In the training data, there are very few observations where this variable is above 15000 AND ...
user avatar
  • 1
0 votes
1 answer
42 views

Access to shape functions in mboost

I am looking into the mboost package in R for creating GAM models. Below is code for creating a very simple GAM model with tree ...
user avatar
  • 1
1 vote
1 answer
39 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
user avatar
1 vote
1 answer
26 views

Boosting usa bootstraping?

I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
user avatar
0 votes
0 answers
22 views

How to fix the tree structure for a tree-based algorithm?

Background Some of our BI analysts and most of our managers are interested in making explainable predictions. One of our colleagues proposed an approach based on individual tree leaves from a tree-...
user avatar
  • 153
0 votes
1 answer
30 views

Linear regression has good performance in validation set despite not meeting the linearity assumption

I have a dataset with about 8000 samples and 18 predictors (16 continuous, 2 categorical). I am trying fit a linear regression, but despite trying multiple transformations, I can't make it meet the ...
user avatar
  • 33
1 vote
1 answer
72 views

XGBoost and how to input feature interactions

I have a dataset with 3 features of interest. Within the boosting (and specifically XGBoost) framework, if I want to account for all possible interactions between the features, does this need to be ...
user avatar
  • 305
0 votes
0 answers
11 views

What if you had a feature column that was the same for every row for a specific date when using XGBoost with time-series data?

Imagine an XGBoost model trying to predict business revenue or performance. Imagine a dataset that looks like this: Date | Business | Revenue | ...more Business properties... | NASDAQ Price | You'd ...
user avatar
  • 209
0 votes
0 answers
11 views

Bias, Variance and Bagging and what it means in relation to representational complexity

Re-looking at some basic ML algorithms bagging comes up along with the Bias-Variance trade off. I am confused on how bagging relates to representational capacity. Based upon the arguments I have read, ...
user avatar
2 votes
0 answers
36 views

Training loss goes back up but validation accuracy continues growing (XGBoost)

Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba. Main ...
user avatar
  • 529
2 votes
1 answer
32 views

Adj. $R^2$ with tree ensembles

Consider tree ensemble methods such as XGBoost, Lightgbm and/or Catboost. Is the adj. $R^2$ a valid metric for tree ensembles? I'm curious because these methods handle factor variables differently. E....
user avatar
  • 218
0 votes
0 answers
35 views

Computational Complexity of Gradient Boosting Decision Tree algorithm

Does anybody have an idea about the computational complexity of GBDT? I have only seen one paper report (Gradient Boosted Decision Trees for High Dimensional Sparse Output). It doesn't seem to be ...
user avatar
0 votes
0 answers
29 views

Aggregate feature importances together

I'm using the feature_importances_ attribute of an XGBoost classifier to plot the feature importances for a classification model. What I'd like to do is group some ...
user avatar
  • 529
0 votes
0 answers
28 views

Sample size problems during Random Search Hyperparameter Tuning

I would like to apply the XGBoost algorithm to my data using the xgbTree method in caret. My ...
user avatar
  • 1
0 votes
0 answers
94 views

How to forecast actual future values using XGBoost?

So I have a solar Irradiation dataset having around 61000+ rows & 2 columns. I have made the model using XGBoost to predict future values. I have split the data in 2 parts train and test and ...
user avatar
0 votes
0 answers
15 views

Regression Trees

Which is better over the other two? Random Forest, Bagging, or Boosting the tree-based method? My understanding is, that even though all three have their own preferred requirements to perform better. ...
user avatar
  • 1
1 vote
1 answer
42 views

LGBM Intuition to Non-technical individuals

How can i explain LGBM to a non-technical person as it involves Trees/Ensembling and much more? Using LGBM for solving a Regression problem and how does it helps in: Better Prediction Feature ...
user avatar
1 vote
0 answers
52 views

XGBoost Feature Importance Changes with Random Seed

Analysis Goal: Identify features that provide an accurate prediction of a binary outcome and also explain how the features are related to the output Data: 72 features and 200 instances. Process: ...
user avatar
0 votes
0 answers
29 views

Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
user avatar
  • 11
0 votes
0 answers
29 views

how to learn on masked data where missing values are predictive of class label

I have a multi-class machine learning problem where some values are missing but not at random. The missing values are indicative of the class label. Specifically the data is masked in such a way that ...
user avatar
  • 474
1 vote
1 answer
30 views

Specifying distribution in GBM

In the GBM package, we can specify which distribution to use that represents our response variable. I have count data and usually, we specify the distribution as Poisson for count data. But, when I ...
user avatar
2 votes
1 answer
114 views

Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?

I have an XGBoost model that I fit on some X data directly out of the box: ...
user avatar
1 vote
0 answers
18 views

Total number of learned weights/parameters in CatBoost model?

I wasn't really able to find this anywhere in the documentation, or after searching through the model object itself (though it must be there somewhere, of course. The ...
user avatar

1
2 3 4 5
27