Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

Filter by
Sorted by
Tagged with
0
votes
0answers
2 views

Performing cross validation for BooST trees

I installed the BooST package, I want to know how to perform cross-validation for a BooST tree regression model. How to do this? Thanks
0
votes
0answers
33 views

Equation 10.2 from The Elements of Statistical Learning. Median of a chi-squared distribution

I'm reading about AdaBoost in the The Elements of Statistical Learning and I don't understand the equation 10.2. Below is an excerpt from the book. The power of AdaBoost to dramatically increase the ...
0
votes
1answer
14 views

Random Forest Parameter Settings for Big Data

I have a big data set (with more than 9,000,000 rows) with 7 features and 1 label. The label is ordinal data. I would like to run a random forest regression. I'm fairly new to random forests so I have ...
1
vote
0answers
15 views

xgboost hyperparameters: interactions that make the model overfit on training set

I am dealing with a classification problem on an unbalanced dataset (positive class is just above 1% of the sample). I did hyperparameter tuning using a train-validation split, and then finally ...
0
votes
0answers
9 views

cross validation and hyperparameter tuning for multiple time series

I have time series data for 2000 products. If I use models like fbprophet or SARIMAX or xgboost then the cross validation needs to be done for 2000 time series data. for a single time series data it ...
0
votes
0answers
19 views

What is the best algorithm to model credit default score for gamification?

Goal: I have a situation that I want to create a model to predict credit default that could handle any missing data for any feature, although my observations don't have any missing data for any ...
1
vote
0answers
36 views

why is gradient boosting machine based algorithm nonparametric although they have a loss function?

I am dealing with very right-skewed distribution of y which seems to follow a tweedie distribution. and I found out it would give a higher performance to change lgbm's loss function to a "tweedie&...
0
votes
0answers
64 views

How is marginalisation being used in the proof of margin-based generalisation error bounds on AdaBoost in Freund & Schapire (2012)?

I am having difficulty in making explicit the use of a standard marginalisation identity in the proof of a lemma relating to a margin-based generalisation error bound on the AdaBoost algorithm. This ...
0
votes
0answers
16 views

Mathematical formalism of Gradient Boosting Decision Trees (GBDT) algorithms

I'm trying to better figure out some formalism behind the Gradient Boosting Decision Trees (GBDT) algorithms. Given a dataset $\mathcal{D}$ and a loss function $L : \mathbb{R}^2 \rightarrow \mathbb{R}$...
0
votes
1answer
27 views

Combining DecisionTreeClassifiers

I have an array of sklearn.tree._classes.DecisionTreeClassifier classifiers that are used in a boosting algorithm, so the final classifier is a weighted sum of these individual trees. The problem is ...
0
votes
1answer
13 views

XGBoost - does it make sense that accuracy decreases as threshold increases?

I'm using XGBoost for a classification problem, and if I need to check how accuracy changes as a function of threshold. As a result, I got that accuracy decreases as the threshold value increases (see ...
0
votes
0answers
14 views

Are tree based models for regression/classification 'endlessly' trainable?

I know this is a relatively simple question that I could answer if I understood trees more. An ANN can be trained indefinitely, especially if it is deep. What other models besides networks have this ...
2
votes
0answers
35 views

Generalisation error bound proof for the Adaboost algorithm, corollary 4.4 in Freund and Schapire (2014)

I am having difficulty with a detail in the proof of generalisation error bounds for the AdaBoost algorithm, and would appreciate some assistance. This is the proof of corollary 4.4 on page 54 of ...
1
vote
0answers
29 views

Difference between ensembling models and using one model as a feature in the second model?

I have 2 models - one is a generative model (a model which assets the parameter of some distribution for each individual object needs to be predicted, and does not learn from any data), and the second ...
1
vote
0answers
11 views

Feature selection using XGBoost, should I add the gains of different CV folds to choose the best N features?

Trying to understand if this is the right approach, without being able to find much through conventional googling. Using XGB, if i have pretty wide data, and I want to choose say, the best 300 (out of ...
3
votes
1answer
122 views

Why is XGBoost so Good? And Boosting/Trees in General?

I'm a bit perplexed after working on a kaggle data science competition for a month now. It's tabular data, all real/floating point numbers, on the order of maybe 100k examples, 100ish features, and a ...
2
votes
1answer
23 views

Exploratory analysis : Regression trees without splitting train-test data

I am analyzing a small dataset of 76 observations and I want to explore how 9 environmental predictors explain my response variable. For this I have decided to use regression trees because I am ...
1
vote
0answers
36 views

Why does classifier (XGBoost) "after PCA" runtime increase compared to "before PCA"

The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
1
vote
0answers
16 views

XGBoost regressor sample weight has negligible impact on performance

I am using XGBoost regressor for a prediction problem. I did a 70/30 split for the available data (around 60K samples) to split training/validation. For the training portion, I used 80% to train the ...
0
votes
0answers
18 views

How does resampling in AdaBoost (exactly) work?

Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
0
votes
0answers
17 views

Unexpected probability distribution from xgboost binary classification

I am testing different a couple of different binary classification models using xgboost to predict likelihood to convert. The difference between the 2 probability distributions shown below are based ...
1
vote
0answers
17 views

Why does non-parametric approach break down when the joint distribution is estimated by a finite data sample?

I am currently reading the paper on Gradient Boosting Machines - J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” Ann. Stat., vol. 29, no. 5, pp. 1189–1232, 2001, doi: 10....
4
votes
1answer
58 views

Intutition of why ensemble learning reduces overfitting?

Can somebody give me a non mathematical intuition why ensemble learning reduce overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen the ...
1
vote
0answers
14 views

Why do Shapley values increase over time?

I calculated the Shapley values (using xgboot package, gbm regression model) of several big actors in the cocoa market and received results which I cannot explain: it seems that Shapley increases (the ...
1
vote
0answers
17 views

Permutation Invariant Tree Gradient Boost Models

Suppose we are dealing with a classification problem whose input is a bag of users $u_i$ and their data, stored as $u_{ij}$ . Each user might have $n$ features like age, gender, description, purchase ...
0
votes
0answers
20 views

R-Squared for count:poisson model

I am trying to develop custom objective function (R-Squared) for count:poisson objective function in XGBoost. I came up with the below skeleton but it does not look like as expected. Not sure whether ...
0
votes
1answer
23 views

If boosting reduce bias why does this work for decision trees as weak learners?

By hastie et al decision trees have low bias and high variance why does boosting work even though bias not being a problem for trees?
0
votes
0answers
30 views

What is the actual Mechanism of Boosting for building the Models?

I couldn't able to find the proper step by step procedure to understand the Boosting Mechanism, how does it build the models and the data used to build it. So, I have gone through tutorials which led ...
0
votes
0answers
34 views

Overfitting in Gradient Boosting

I have a very unbalanced dataset(99.8% negative,0.2% positive) with approximately 60 variables. I removed somewhere around 40 variables based on the variance inflation factor. Then I used SMOTE to ...
0
votes
0answers
17 views

Is L2 Boosting just a variant of Gradient Boosting?

My understanding is that L2 Boosting is using functional gradient descent. Does this means it is one variant of Gradient Boosting?
1
vote
2answers
38 views

feature importance aggregation

I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. ...
0
votes
0answers
20 views

Likelihood function 'sampled' from the posterior, uncertainty quantification in gradient boosting models

I was reading the following paper https://arxiv.org/abs/2006.10562 that is about estimating predictive uncertainty by using well known gradient boosting models like xgboost, catboost, LightGBM, etc. ...
0
votes
0answers
21 views

How to determine number of boost for calculating Krippendorff’s Alpha?

I'm using the SPSS macro for computing Krippendorff’s Alpha, developed by Andrew F. Hayes & Klaus Krippendorff. One argument needed is the units of bootstrapping, and I did not find formula for ...
1
vote
1answer
15 views

nested cross validaiton vs train-test split

I am trying to understand the main benefits of conducting a nested cross-validation compared to a simpler train-test split. Let us say I would like to build a prediction model. I initially split my ...
0
votes
0answers
30 views

Why is the Hessian of RegressionL1Loss set to 1 in LightGBM

I'm reading this code snippet related to RegressionL1loss implementation in LightGBM ...
0
votes
0answers
4 views

Decision function to draw conclusion from two separate models

So I have trained two separate classifiers using sklearn's built-in Gradient Boosting Classifier. One of the classifier is responsible for classifying four classes(0, 1, 4, 6) while the other one is ...
0
votes
1answer
39 views

How to find key influencers variables in a formula Y=X1+X2+...+Xn?

I've been trying to find which are the key clients in a sales trend. I have the following dataset which follows the formula Y(t) = X1+X2+...+Xn whereas each Xi is ...
1
vote
1answer
60 views

Is my xgboost model overfitting? (timeseries)

I am very skeptical looking at this, I haven't used the xgboost library that much before, maybe someone can help me out. The errors and the predictions on my test ...
1
vote
0answers
41 views

XGBoost Cross Validation - Baseline model performs better on validation data but it performs well below the training performance

I appreciate it if someone guides me on the following situation: I'm trying to decide what parameter set to choose for my XGBClassifier. The dataset has roughly 200'...
0
votes
0answers
31 views

python: xgboost parameters overfitting

I am using the xgboost regression algorithm to predict a continuous variable. after splitting the data between train and test, I kept changing the xgb parameters to obtain the best possible predictive ...
0
votes
0answers
14 views

Boosting models performing worse than bagging models

I've noticed that, sometimes, models that are based on boosting, such as Gradient Boosting, show worse performance than pretty similar models, but based on bagging. For example, Random Forest (bagging ...
0
votes
0answers
16 views

Hyperparameter selection with cross validation may not work well when training on entire train set

Imagine a typical procedure to train a binary classifier. Assume a 4-fold cross validation (CV) scenario with 40,000 training samples and 200 features to select the best hyperparameters for a ...
1
vote
1answer
43 views

Choice of a loss function

Im running an xgboost model to try and find important predictors for a disease from a list of almost 1000 covariates. The prevalence of the disease in my cohort is about 10%. Given the imbalance data, ...
2
votes
0answers
41 views

Classification algorithm find closest observations

I am currently using Light GBM (Gradient Boosting) in Python, but this question can apply to other classification algorithms. In order to improve my model's explainability I would like to be able to ...
2
votes
1answer
43 views

nested CV for parameter importance

I was wondering whether you can help me sort out some confusion I have about using nested cross validation (i.e. inner + outer cv loops) in a ML analysis. I am happy to share some code, but I think ...
1
vote
0answers
16 views

Static ML model or Time-Series? How to model/predict a Binary target when I have time variant features but most features are constant?

I have been working with Real World data from patients. I have a dataset with information about 10million patients; Collected over a span of varying duration (5 to 20 years). What I am predicting is ...
0
votes
0answers
9 views

Any issues with conducting stratafied train/test splitting based on the distribution of a categorical predictor?

I am building a xgboost regressor for a dataset that includes a categorical feature with a very large number of levels (on average, each level has an observation frequency of only about .2%). However, ...
3
votes
0answers
69 views

Random Forest vs Gradient Boosting out of distribution

I'm working on a classification task where I have data from a certain company for years between 2017 and 2020. Trying to train different models (Random Forest, XgBoost, LightGBM, Catboost, Explainable ...
1
vote
2answers
88 views

Calibration of a few binary classifiers is not perfect - why?

I am working on a binary classifier using LightGBM. I try to see the results of the classifiers when changing the costs of false positives and false negatives, still working on the same training and ...
1
vote
1answer
66 views

What's the purpose of learning rate in sklearn AdaBoost implementation

We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner. Conceptually, ...

1
2 3 4 5
26