Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
1 vote
0 answers
11 views

Random Forests- Out of Bag Error Calculation

I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
user avatar
0 votes
0 answers
12 views

Are tree based algorithms weak at correctly predicting an outcome for a given variable range, if there is sparse data for that variable range?

I am using XGBoost as a classifier, and one of my important variables has values ranging from 500 to 20000. In the training data, there are very few observations where this variable is above 15000 AND ...
user avatar
  • 1
0 votes
0 answers
7 views

Random Forest - boosting output [closed]

what is random forest boosting output for even number of decision trees with balanced output? eg. DT1 - 1, DT2 - 0, DT3 - 1, DT4 - 0, DT5 - 1, DT6 - 0
user avatar
  • 1
0 votes
1 answer
38 views

Access to shape functions in mboost

I am looking into the mboost package in R for creating GAM models. Below is code for creating a very simple GAM model with tree ...
user avatar
  • 1
1 vote
1 answer
35 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
user avatar
1 vote
1 answer
20 views

Boosting usa bootstraping?

I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
user avatar
0 votes
0 answers
21 views

How to fix the tree structure for a tree-based algorithm?

Background Some of our BI analysts and most of our managers are interested in making explainable predictions. One of our colleagues proposed an approach based on individual tree leaves from a tree-...
user avatar
  • 153
0 votes
1 answer
28 views

Linear regression has good performance in validation set despite not meeting the linearity assumption

I have a dataset with about 8000 samples and 18 predictors (16 continuous, 2 categorical). I am trying fit a linear regression, but despite trying multiple transformations, I can't make it meet the ...
user avatar
  • 33
1 vote
1 answer
27 views

XGBoost and how to input feature interactions

I have a dataset with 3 features of interest. Within the boosting (and specifically XGBoost) framework, if I want to account for all possible interactions between the features, does this need to be ...
user avatar
  • 185
0 votes
0 answers
7 views

What if you had a feature column that was the same for every row for a specific date when using XGBoost with time-series data?

Imagine an XGBoost model trying to predict business revenue or performance. Imagine a dataset that looks like this: Date | Business | Revenue | ...more Business properties... | NASDAQ Price | You'd ...
user avatar
  • 209
0 votes
0 answers
11 views

Bias, Variance and Bagging and what it means in relation to representational complexity

Re-looking at some basic ML algorithms bagging comes up along with the Bias-Variance trade off. I am confused on how bagging relates to representational capacity. Based upon the arguments I have read, ...
user avatar
1 vote
0 answers
20 views

Training loss goes back up but validation accuracy continues growing (XGBoost)

Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba. Main ...
user avatar
  • 509
2 votes
1 answer
32 views

Adj. $R^2$ with tree ensembles

Consider tree ensemble methods such as XGBoost, Lightgbm and/or Catboost. Is the adj. $R^2$ a valid metric for tree ensembles? I'm curious because these methods handle factor variables differently. E....
user avatar
  • 218
0 votes
0 answers
21 views

Computational Complexity of Gradient Boosting Decision Tree algorithm

Does anybody have an idea about the computational complexity of GBDT? I have only seen one paper report (Gradient Boosted Decision Trees for High Dimensional Sparse Output). It doesn't seem to be ...
user avatar
0 votes
0 answers
22 views

Aggregate feature importances together

I'm using the feature_importances_ attribute of an XGBoost classifier to plot the feature importances for a classification model. What I'd like to do is group some ...
user avatar
  • 509
0 votes
0 answers
25 views

Sample size problems during Random Search Hyperparameter Tuning

I would like to apply the XGBoost algorithm to my data using the xgbTree method in caret. My ...
user avatar
  • 1
0 votes
0 answers
25 views

How to forecast actual future values using XGBoost?

So I have a solar Irradiation dataset having around 61000+ rows & 2 columns. I have made the model using XGBoost to predict future values. I have split the data in 2 parts train and test and ...
user avatar
0 votes
0 answers
15 views

Regression Trees

Which is better over the other two? Random Forest, Bagging, or Boosting the tree-based method? My understanding is, that even though all three have their own preferred requirements to perform better. ...
user avatar
  • 1
1 vote
1 answer
36 views

LGBM Intuition to Non-technical individuals

How can i explain LGBM to a non-technical person as it involves Trees/Ensembling and much more? Using LGBM for solving a Regression problem and how does it helps in: Better Prediction Feature ...
user avatar
1 vote
0 answers
40 views

XGBoost Feature Importance Changes with Random Seed

Analysis Goal: Identify features that provide an accurate prediction of a binary outcome and also explain how the features are related to the output Data: 72 features and 200 instances. Process: ...
user avatar
0 votes
0 answers
20 views

Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
user avatar
  • 11
0 votes
0 answers
18 views

how to learn on masked data where missing values are predictive of class label

I have a multi-class machine learning problem where some values are missing but not at random. The missing values are indicative of the class label. Specifically the data is masked in such a way that ...
user avatar
  • 474
1 vote
1 answer
26 views

Specifying distribution in GBM

In the GBM package, we can specify which distribution to use that represents our response variable. I have count data and usually, we specify the distribution as Poisson for count data. But, when I ...
user avatar
2 votes
1 answer
72 views

Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?

I have an XGBoost model that I fit on some X data directly out of the box: ...
user avatar
1 vote
0 answers
15 views

Total number of learned weights/parameters in CatBoost model?

I wasn't really able to find this anywhere in the documentation, or after searching through the model object itself (though it must be there somewhere, of course. The ...
user avatar
1 vote
1 answer
28 views

Generalization of model performance (AUC) and tuning of a catboost classifier

I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome. Wouldn't it be better to reguralize until validation error equals ...
user avatar
1 vote
1 answer
48 views

Loess line interpretation

I'm sorry for this noob question, but I'm following a practical to draw a plot of Boston Housing data set after using Gradient Boosting Machine to train the data, but I don't understand how to ...
user avatar
1 vote
1 answer
52 views

Solve Local Minima Problem through Averaging

I am using neural networks (created in R: neuralnet) to predict county-level food insecurity. I want to use Olden's connection weight approach to analyze relative ...
user avatar
  • 29
0 votes
0 answers
15 views

Does boosting help select better features compared to bagging?

I am working on a binary classification problem using traditional algorithm and neural networks. with 977 records, my class proportion is 77:23 Currently, I am doing the below steps a) Identify the ...
user avatar
  • 1,748
2 votes
0 answers
42 views

Diverging Gradient Boosting Optimization with custom loss

I have a supervised learning problem (regression) with $m$ input features $X=(x_1, ..., x_m)$ with output $y$. I want to predict a "multiplicative correction factor" $\hat \alpha(X)$ such ...
user avatar
  • 1,486
6 votes
3 answers
432 views

XGBoost Classifier not capturing extreme probabilities

I'm using XGBoost for a binary classification task—trying to predict whether team A will beat team B given the score of the game and the time left. I know for certain score-time combinations, the ...
user avatar
0 votes
0 answers
9 views

Predicitions from conditional inference forests for ordinal responses

How do I get class predictions from a conditional inference forest (utilizing ordinal regression trees) for an ordinal response? By majority vote or average or to classify into the most likely class? ...
user avatar
  • 1
0 votes
0 answers
32 views

How to compare cross-validation results against test results (XGBoost model)?

I am building a gradient boosting regression model with XGBOOST and testing different versions of the model by adding or modifying some features. The target variable is a skewed continuous variable. I ...
user avatar
  • 1
0 votes
0 answers
70 views

How to Determine Gradient and Hessian for Custom Xgboost Functions

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
user avatar
  • 51
0 votes
1 answer
37 views

How to determine if the binary response was just randomly assigned and not due to any of the predictors

I am practicing fitting models to data sets that I could find on Kaggle but I don't know how these data sets were generated. I remember there was a data set I played around with in order to fit a ...
user avatar
1 vote
1 answer
46 views

How to calibrate an XGBoost classifier which has been trained on a sampled dataset?

I have trained my xgboost binary classifier on a dataset which does not represent the true proportion of positive over negative observations of the population. The ...
user avatar
  • 11
1 vote
1 answer
35 views

can tree-based models extrapolate with categorical independent variables

I am aware of the fact that tree-based (machine learning) models struggle to extrapolate - see regression example here. I am only familiar with the concept of extrapolation for numeric independent ...
user avatar
  • 1,306
1 vote
0 answers
58 views

Predicted Probability with XGBClassifier ranging only from 0.48 to 0.51 for either class

Why does my XGBClassifier predicts probability only from 0.48 to 0.51 for either class? I'm very new to XGBoost, so any suggestions are greatly appreciated! Here's ...
user avatar
  • 111
0 votes
1 answer
46 views

when to choose predictive power of a model over explainability?

I am currently working on a binary classification with 1000 records. class distribution is 75:25 I tried both logistic and random forests, My results are better for random forests. However, when I try ...
user avatar
  • 1,748
0 votes
0 answers
9 views

Gradient Boosting data

I understand gradient boosting as sequential modelling based on residuals from the last model. And I have read in various resources, articles etc that in every sequential model we should be focusing ...
user avatar
  • 143
2 votes
2 answers
107 views

How to deal with random parameters in MLOps

I have a XGB model ready to go to production, in validation I discovered that the random seed makes reasonable difference in the performance of the model, which is pretty good, but for some seeds it's ...
user avatar
0 votes
1 answer
33 views

Using a feature as a target denominator

Could you please tell me what (bad) can happen if I use the same feature as the denominator in the target feature and as the predictor in a boosting regression? I think I should exclude it from the ...
user avatar
1 vote
0 answers
222 views

Formal steps for gradient boosting with softmax and cross entropy loss function

Consider some data $\{(x_i,y_i)\}^n_{i=1}$ and a differentiable loss function $\mathcal{L}(y,F(x))$ and a multiclass classification problem which should be solved by a gradient boosting algorithm. ...
user avatar
  • 218
0 votes
2 answers
45 views

XGBoost heavily overfitting when containing the minimum/maximum of a variable?

I've been building an XGBoost Regressor model with some good success. Currently, the training accuracy is 68% and testing 66% - indicating some, but not too much, overfitting. However, I've noticed ...
user avatar
0 votes
0 answers
18 views

xgboost - difference between XGBClassifier.feature_importances_ and XGBClassifier.get_booster().get_fscore()

What is the difference between get_fscore() and feature_importances? Both are explained as feature importance but the importance values are different. ...
user avatar
  • 755
0 votes
0 answers
42 views

What's the difference between combined pos_bagging_fraction and neg_bagging_fraction vs is_unbalance vs scale_pos_weight in LightGBM?

Let's suppose a binary classification task and an unbalanced dataset (10% of positive records). I am using LightGBM and would like to better understand the difference between the combined ...
user avatar
  • 33
0 votes
0 answers
27 views

Categorical Variables in Clustering V/S Decision Trees

I read that most basic clustering algorithms cannot be used when we have qualitative predictors in the dataset. However, decision trees can be constructed in the presence of qualitative predictors. I ...
user avatar
1 vote
1 answer
38 views

Why do gradient boosting algorithms mostly use trees?

Why do gradient boosting algorithms mostly use trees? Is there any logic in this? (in XGBoost and in boosting which in sklearn library uses trees, not other algorithms).
user avatar
  • 23
0 votes
0 answers
28 views

Question about using ML (xgboost) for timeseries

I want to train an ML model with 3 input parameters 1 output using Xgboost regressor. My output data is time-dependent meaning that I have the values of output parameters at 500 time steps. Also, I ...
user avatar
  • 151
3 votes
0 answers
25 views

should trees in an ensemble be trained on samples of the same size?

I know that if bootstrap=True, then "for each tree, N samples are drawn randomly with replacement from the training set and the tree is built on this new ...
user avatar

1
2 3 4 5
27