# Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

1,305 questions
Filter by
Sorted by
Tagged with
1 vote
11 views

### Random Forests- Out of Bag Error Calculation

I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
12 views

### Are tree based algorithms weak at correctly predicting an outcome for a given variable range, if there is sparse data for that variable range?

I am using XGBoost as a classifier, and one of my important variables has values ranging from 500 to 20000. In the training data, there are very few observations where this variable is above 15000 AND ...
• 1
7 views

### Random Forest - boosting output [closed]

what is random forest boosting output for even number of decision trees with balanced output? eg. DT1 - 1, DT2 - 0, DT3 - 1, DT4 - 0, DT5 - 1, DT6 - 0
38 views

### Access to shape functions in mboost

I am looking into the mboost package in R for creating GAM models. Below is code for creating a very simple GAM model with tree ...
1 vote
35 views

### XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: $$L^{(t)} \text{: cross entropy loss function.}$$ ...
• 13
1 vote
20 views

### Boosting usa bootstraping?

I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
• 125
21 views

### How to fix the tree structure for a tree-based algorithm?

Background Some of our BI analysts and most of our managers are interested in making explainable predictions. One of our colleagues proposed an approach based on individual tree leaves from a tree-...
• 153
28 views

### Linear regression has good performance in validation set despite not meeting the linearity assumption

I have a dataset with about 8000 samples and 18 predictors (16 continuous, 2 categorical). I am trying fit a linear regression, but despite trying multiple transformations, I can't make it meet the ...
• 33
1 vote
27 views

### XGBoost and how to input feature interactions

I have a dataset with 3 features of interest. Within the boosting (and specifically XGBoost) framework, if I want to account for all possible interactions between the features, does this need to be ...
• 185
7 views

### What if you had a feature column that was the same for every row for a specific date when using XGBoost with time-series data?

Imagine an XGBoost model trying to predict business revenue or performance. Imagine a dataset that looks like this: Date | Business | Revenue | ...more Business properties... | NASDAQ Price | You'd ...
• 209
11 views

### Bias, Variance and Bagging and what it means in relation to representational complexity

Re-looking at some basic ML algorithms bagging comes up along with the Bias-Variance trade off. I am confused on how bagging relates to representational capacity. Based upon the arguments I have read, ...
• 151
1 vote
20 views

### Training loss goes back up but validation accuracy continues growing (XGBoost)

Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba. Main ...
• 509
32 views

### Adj. $R^2$ with tree ensembles

Consider tree ensemble methods such as XGBoost, Lightgbm and/or Catboost. Is the adj. $R^2$ a valid metric for tree ensembles? I'm curious because these methods handle factor variables differently. E....
• 218
21 views

### Computational Complexity of Gradient Boosting Decision Tree algorithm

Does anybody have an idea about the computational complexity of GBDT? I have only seen one paper report (Gradient Boosted Decision Trees for High Dimensional Sparse Output). It doesn't seem to be ...
22 views

### Aggregate feature importances together

I'm using the feature_importances_ attribute of an XGBoost classifier to plot the feature importances for a classification model. What I'd like to do is group some ...
• 509
25 views

### Sample size problems during Random Search Hyperparameter Tuning

I would like to apply the XGBoost algorithm to my data using the xgbTree method in caret. My ...
25 views

### How to forecast actual future values using XGBoost?

So I have a solar Irradiation dataset having around 61000+ rows & 2 columns. I have made the model using XGBoost to predict future values. I have split the data in 2 parts train and test and ...
15 views

### Regression Trees

Which is better over the other two? Random Forest, Bagging, or Boosting the tree-based method? My understanding is, that even though all three have their own preferred requirements to perform better. ...
1 vote
36 views

### LGBM Intuition to Non-technical individuals

How can i explain LGBM to a non-technical person as it involves Trees/Ensembling and much more? Using LGBM for solving a Regression problem and how does it helps in: Better Prediction Feature ...
• 101
1 vote
40 views

### XGBoost Feature Importance Changes with Random Seed

Analysis Goal: Identify features that provide an accurate prediction of a binary outcome and also explain how the features are related to the output Data: 72 features and 200 instances. Process: ...
20 views

### Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
• 11
18 views

### how to learn on masked data where missing values are predictive of class label

I have a multi-class machine learning problem where some values are missing but not at random. The missing values are indicative of the class label. Specifically the data is masked in such a way that ...
• 474
1 vote
26 views

### Specifying distribution in GBM

In the GBM package, we can specify which distribution to use that represents our response variable. I have count data and usually, we specify the distribution as Poisson for count data. But, when I ...
72 views

### Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?

I have an XGBoost model that I fit on some X data directly out of the box: ...
• 61
1 vote
15 views

### Total number of learned weights/parameters in CatBoost model?

I wasn't really able to find this anywhere in the documentation, or after searching through the model object itself (though it must be there somewhere, of course. The ...
1 vote
28 views

### Generalization of model performance (AUC) and tuning of a catboost classifier

I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome. Wouldn't it be better to reguralize until validation error equals ...
• 35
1 vote
48 views

### Loess line interpretation

I'm sorry for this noob question, but I'm following a practical to draw a plot of Boston Housing data set after using Gradient Boosting Machine to train the data, but I don't understand how to ...
1 vote
52 views

### Solve Local Minima Problem through Averaging

I am using neural networks (created in R: neuralnet) to predict county-level food insecurity. I want to use Olden's connection weight approach to analyze relative ...
• 29
15 views

### Does boosting help select better features compared to bagging?

I am working on a binary classification problem using traditional algorithm and neural networks. with 977 records, my class proportion is 77:23 Currently, I am doing the below steps a) Identify the ...
• 1,748
42 views

### Diverging Gradient Boosting Optimization with custom loss

I have a supervised learning problem (regression) with $m$ input features $X=(x_1, ..., x_m)$ with output $y$. I want to predict a "multiplicative correction factor" $\hat \alpha(X)$ such ...
• 1,486
432 views

### XGBoost Classifier not capturing extreme probabilities

I'm using XGBoost for a binary classification task—trying to predict whether team A will beat team B given the score of the game and the time left. I know for certain score-time combinations, the ...
9 views

### Predicitions from conditional inference forests for ordinal responses

How do I get class predictions from a conditional inference forest (utilizing ordinal regression trees) for an ordinal response? By majority vote or average or to classify into the most likely class? ...
• 1
32 views

### How to compare cross-validation results against test results (XGBoost model)?

I am building a gradient boosting regression model with XGBOOST and testing different versions of the model by adding or modifying some features. The target variable is a skewed continuous variable. I ...
• 1
70 views

### How to Determine Gradient and Hessian for Custom Xgboost Functions

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
• 51
37 views

### How to determine if the binary response was just randomly assigned and not due to any of the predictors

I am practicing fitting models to data sets that I could find on Kaggle but I don't know how these data sets were generated. I remember there was a data set I played around with in order to fit a ...
• 165
1 vote
46 views

### How to calibrate an XGBoost classifier which has been trained on a sampled dataset?

I have trained my xgboost binary classifier on a dataset which does not represent the true proportion of positive over negative observations of the population. The ...
• 11
1 vote
35 views

### can tree-based models extrapolate with categorical independent variables

I am aware of the fact that tree-based (machine learning) models struggle to extrapolate - see regression example here. I am only familiar with the concept of extrapolation for numeric independent ...
• 1,306
1 vote
58 views

### Predicted Probability with XGBClassifier ranging only from 0.48 to 0.51 for either class

Why does my XGBClassifier predicts probability only from 0.48 to 0.51 for either class? I'm very new to XGBoost, so any suggestions are greatly appreciated! Here's ...
• 111
46 views

### when to choose predictive power of a model over explainability?

I am currently working on a binary classification with 1000 records. class distribution is 75:25 I tried both logistic and random forests, My results are better for random forests. However, when I try ...
• 1,748
9 views

### Gradient Boosting data

I understand gradient boosting as sequential modelling based on residuals from the last model. And I have read in various resources, articles etc that in every sequential model we should be focusing ...
• 143
107 views

### How to deal with random parameters in MLOps

I have a XGB model ready to go to production, in validation I discovered that the random seed makes reasonable difference in the performance of the model, which is pretty good, but for some seeds it's ...
33 views

### Using a feature as a target denominator

Could you please tell me what (bad) can happen if I use the same feature as the denominator in the target feature and as the predictor in a boosting regression? I think I should exclude it from the ...
1 vote
222 views

### Formal steps for gradient boosting with softmax and cross entropy loss function

Consider some data $\{(x_i,y_i)\}^n_{i=1}$ and a differentiable loss function $\mathcal{L}(y,F(x))$ and a multiclass classification problem which should be solved by a gradient boosting algorithm. ...
• 218
45 views

### XGBoost heavily overfitting when containing the minimum/maximum of a variable?

I've been building an XGBoost Regressor model with some good success. Currently, the training accuracy is 68% and testing 66% - indicating some, but not too much, overfitting. However, I've noticed ...
18 views

### xgboost - difference between XGBClassifier.feature_importances_ and XGBClassifier.get_booster().get_fscore()

What is the difference between get_fscore() and feature_importances? Both are explained as feature importance but the importance values are different. ...
• 755
42 views

### What's the difference between combined pos_bagging_fraction and neg_bagging_fraction vs is_unbalance vs scale_pos_weight in LightGBM?

Let's suppose a binary classification task and an unbalanced dataset (10% of positive records). I am using LightGBM and would like to better understand the difference between the combined ...
• 33
27 views

### Categorical Variables in Clustering V/S Decision Trees

I read that most basic clustering algorithms cannot be used when we have qualitative predictors in the dataset. However, decision trees can be constructed in the presence of qualitative predictors. I ...
• 187
1 vote
38 views

### Why do gradient boosting algorithms mostly use trees?

Why do gradient boosting algorithms mostly use trees? Is there any logic in this? (in XGBoost and in boosting which in sklearn library uses trees, not other algorithms).
• 23