Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

0
votes
0answers
10 views

Probability calibration from LightGBM model with class imbalance

I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
0
votes
0answers
12 views

Why is the sqrt(n_features) the default maximum number of features for the best split in RandomForestClassifier? [duplicate]

Why does sklearn.ensemble.RandomForestClassifier references have $\sqrt{n}$ in the max_features implementation and why does randomForest in R seem to have the same $\sqrt{n}$ default? I am looking for ...
0
votes
0answers
7 views

Can I use Gradient Boosting Classier to determine feature importance without worrying about precision, recall and accuracy?

My boss is interested in understanding how certain actions improve user retention WoW. I decided to build a GBDT model to assess those features. My question is: Does accuracy, precision or recall ...
0
votes
0answers
16 views

Question about generalized boosted models (GBM) with Poisson assumption

I'm trying to learn how to use GBM R-package. I'm trying to model count data with the gradient boosting, so I'm using the Poisson distribution option in GBM. According to this PDF (page 12) http://...
0
votes
0answers
16 views

How can I run a decision tree algorithm with a specific hierarchy of variables and with many missing values?

I asked students in learning groups what their biggest learning problem was "today" for each learner. The biggest problem could either be "motivational" (=motivation problem) or cognitive (="knowledge ...
0
votes
0answers
6 views

Lars, glmboost & predict function in R

Below you can find a simple Monte Carlo simulation with 50 iterations. My goal is to print the mean-squared prediction error (MSPE) for lasso predictor and boosting with componentwise least squares. ...
0
votes
1answer
28 views

Interpreting RMSE of log-values

I am modelling a regression with a GBM and evaluate by RMSE. My model input & target is log-transformed which results in an RMSE that is also on log-scale. How can i interpret this in an ...
0
votes
0answers
38 views

Does hyperparameters of machine learning algorithms necessarily change performance? [closed]

I'm working on dataset (1100 rows and 40 000 features) with target variable very unbalanced (5% of positives). Train sample represents 80% of rows and test sample 20 %. I try to compare many ...
1
vote
1answer
58 views

Why the discrepancy between predict.xgb.Booster & xgboostexplainer prediction contributions?

One way to explain individual predictions of an xgb classifier is to calculate contributions of each feature. To my knowledge there are two packages in R that can do this for you automatically. In the ...
7
votes
2answers
1k views

Is it possible to combine predictions to improve overall prediction quality?

This is a binary classification problem. The metric that is being minimised is the log loss ( or cross entropy ). I also have an accuracy number, just for my information. It is a large, very ...
0
votes
0answers
8 views

What causes a high testing deviance vs. training deviance in a gradient boosting classifier?

My main goal is to classify multi-class data using supervised learning. Currently, I am looking into GradientBoostingClassifier as the estimator. I want to make sure I am selecting the model ...
1
vote
0answers
16 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
0
votes
1answer
28 views

focusing on hard examples in neural networks, like in gradient boosting?

gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar ...
0
votes
0answers
10 views

gradient with respect to totally different things in neural networks vs gradient boosting?

I'm confused about the usage of gradients in NN vs. GBM. Is is correct to say that the gradient is with respect to totally different things? my understanding is that: in NN (I'm following ...
0
votes
1answer
22 views

Cross-validation with Boosting Trees (do I need 4 sets?)

Normally, you have train, validation and test sets for training, tuning (hyperparameters) and finally evaluating a machine-learning model. If we use cross-validation, then we can effectively have only ...
0
votes
0answers
24 views

Counterfactual prediction with machine learning, sales data

I have a dataset from a supermarket with around 10 thousand products. The data has daily quantities and prices and discount information (whether the product had a discount and the size of the discount ...
1
vote
0answers
21 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
1
vote
0answers
13 views

Boosted Regression Trees: Zero discrimination and calibration scores

I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the $cv.statistics What would be the reason ...
0
votes
0answers
12 views

Scale response variable y in random forest or gradient boosted trees for regression == scale prediction?

Suppose we are fitting a random forest or gradient boosted tree model for regression on y. We first fit the model. Later on, we realized we need to fit y at another different scale, for example, a <...
0
votes
0answers
13 views

Results from BRTs and GAMMs differ

I have recently run some biological density data through a boosted regression trees (BRTs) and a generalised additive mixed-effects model (GAMM) to find the best environmental predictor of long-term ...
0
votes
0answers
15 views

how to learn a GradientBoostingClassifier model with with a fixed precision?

I would to have a final model with a fixed precision (say 0.75) and improve the recall as much as possible. How can i do that sklearn ?
1
vote
2answers
38 views

How NULLs in numerical variables are treated in tree-based models?

I understand that in tree-based models (CART, Gradient boosted trees, etc.), NULLs (i.e., NaN) in categorical variables can be treated as a separated category, while making node splits. However, how ...
1
vote
2answers
30 views

I am learning to do stacking. I want to know what will the input be to level 2 classifiers [closed]

In classification/regression problems, say if we use five different base classifiers, we get 5 predictions for each example. What would be the input to the second level classifiers?
0
votes
0answers
8 views

What a convex Precision-Recall curve means for training dataset?

Situation I have trained a GBDT model(gradient-boosted decision tree, a tree ensemble model) with a training dataset, and when I calculate PR curve on the same training set, it looks convex: For ...
0
votes
1answer
40 views

Is Gradient Boosting Regression Tree able to learn linear models

Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in [0,1]$. Assume one use gbdt to learn this linear model. And if ...
1
vote
0answers
14 views

How to deal with random state affecting feature selection? (gradient-boosted trees)

I'm dealing with an imbalanced classification problem and I'd like to use feature importance from gradient boosting decision trees for recursive feature elimination in order to get rid of redundant ...
0
votes
0answers
11 views

Normality of residuals in Boosting

I'm very new to the machine learning field and am trying to apply boosting to my models to improve its predictive ability. I was doing some reading into tree-based approaches and was wondering whether ...
1
vote
0answers
18 views

Boosted regression trees update of residuals

In my book on statistical learning, an algorithm for boosting for regression trees is described. They have the main step of the algorithm as: $\hat{f}(x) \leftarrow f(x) + \lambda \hat{f}^b(x)$, ...
4
votes
1answer
101 views

On what types of datasets do tree-based models not do well?

Are there examples where splitting on the best feature/threshold combination is not actually the best way to split the tree, and that better results could be got by choosing a different feature but ...
1
vote
0answers
8 views

Boosted regression trees - clarification of algorithm step

In my statistic learning textbook, there is an algorithm, "Boosting for Regression Trees". As step 1 in the algorithm, it is said to set $\hat{f}(x) = 0$ and $r_i = y_i$ for all $i$ in the training ...
1
vote
0answers
17 views

Decision tree / boosting tree in Tensorflow?

When I was looking for some way to break the bottleneck of memory limitation in xgboost, I found there are some boosting tree algorithm that is implemented in Tensorflow such as here. I have some ...
3
votes
0answers
23 views

Is Gradiant Boosting a generalization of Adaboost?

I read somewhere that Gradiant boosting is a generalization of Adaboost. However, I cannot see why. Can Anyone elaborate?
1
vote
0answers
43 views

Machine Learning on Extremely Low Signal Data

I have terabytes of data with an extremely low signal to noise ratio, with the following characteristics: The relationship between the features and the response variable can change over time I'm ...
0
votes
1answer
31 views

How to correctly retrain model using all data, after cross-validation with early stopping

I have a classification task that doesn't have loads and loads of data, so I'd like to make the most of the data. I have a boosting model and I've performed 5-fold CV, using the validation fold for ...
1
vote
2answers
67 views

Which formula for GBM is correct?

I am trying to write a simple GBM simulator. Unfortunately, the task has turned rather difficult. The first approach I looked into was the most obvious. I could use the analytic solution for the GBM ...
1
vote
0answers
103 views

XGBoost Poisson Objective Function When Data is Over-dispersed [closed]

I am modeling very over-dispersed count data with the goal of prediction. The data is not zero inflated (there are no zeros), but there are a lot of values of 1. ...
0
votes
0answers
34 views

What is the interpretation of GBM model output via caret train function?

I am running GBM method for a Classification problem in Caret R. With Verbose=T and Resampling method as "repeatedcv" with 5-folds & 5-repeats, I get the following values printed in console: ...
4
votes
1answer
91 views

Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
3
votes
0answers
103 views

About Partial dependence for Poisson GLM

Can someone tell me what would be the expression for calculating the partial dependence on a GLM model with family specified as Poisson? From applying Friedman partial dependence estimation ...
0
votes
0answers
24 views

Data standardization or normalization in GBDT [duplicate]

Is it necessary to do data normalization(standardization) before using gbdt?what effect does it have if I don't do that proprecessing?
2
votes
1answer
83 views

How to constrain gradient boosting predictions to be non-negative?

I have a regression problem with the target always nonnegative, I used gbdt as the model, but sometimes the model outputs negative prediction value.Is there any way to output nonnegative value using ...
0
votes
0answers
13 views

Computational Optimization of Cross-Validation in Training Models

TL;DR: If I run a 5-fold cross-validation which assigns 1 fold per core when my CPU only has 4 cores, is the 5th as costly (with respect to computation time) as the first 4 folds? Context: I am ...
2
votes
1answer
21 views

Decision Tree - What to avoid first a Skewed dataset or reduce too much the number of bins

So I have a dataset with a categorical column rather skewed. Lets imagine something like this: ...
1
vote
0answers
26 views

How do Gradient-boosted trees cope with new categories after training?

I'm looking at using a Gradient-boosted tree model to predict categories for a dataset. The data has has multiple categorical variables which have high cardinality. I've converted the dataset into ...
2
votes
1answer
43 views

Get the number of weak learner - ebmc package of R - implementing class imbalance RusBoost on my dataset

I'm new to class imbalance and applying class imbalance technique 'RusBoost' on my dataset. I'm using ebmc package from R. I'm having difficulties to get its arguements values, as per the ...
0
votes
0answers
30 views

Gradient Boosted Regression - decide number of trees?

By adding arbitrarily many trees, seems like the $R^2$ value can be as close to 1.0 as we want. This doesn't seem correct. How do we determine the optimal number of trees? Should I use a form of ...
1
vote
1answer
35 views

Find a weak learner in Boosting

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, PDF source Question 1: Why find the weak learner by the formula ...
0
votes
0answers
41 views

How to improve GBM performance

I'm trying to model insurance losses, with a Tweedie distribution. I have a data set of about 40 million records, and over 100 independent variables. My response variable is "loss", I take the log of ...
0
votes
0answers
34 views

Residuals' spatial autocorrelation in Boosted Regression Trees after correcting for it

I am running boosted regression trees (BRT) in R, with the package dismo and I have included a predictor (residual autocovariate) that, in theory, correct for spatial autocorrelation, following a ...
1
vote
1answer
69 views

How does gbm for classification work?

I have a got a fair idea about how it works in regression where each successive decision tree tries to predict the residual (negative gradient for loss function) and the predicted value gets added to ...