Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [boosting]

A family of algorithms combining weakly predictive models into a strongly predictive model. The most common approach is called gradient boosting, and the most commonly used weak models are classification/regression trees.

0
votes
0answers
10 views

Matching Pursuit & Boosting: Exponential Convergence?

In the paper by Bühlmann BOOSTING FOR HIGH-DIMENSIONAL LINEAR MODELS, he introduces an algorithm, called componentwise linear least squares, and relates it to the Matching Pursuit algorithm by Mallat &...
2
votes
0answers
13 views

In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
3
votes
1answer
86 views

Please correct my assumption on how regression trees work

I'm trying to understand how regression trees work, I've been experimenting with catboost and xgboost in python, and I'm getting results which I don't expect, can someone please clarify (and apologies ...
-2
votes
0answers
10 views

H2O GBM POJO shows error? [closed]

Hi I am trying to develop a webapp using H2O pojo. But I am getting the following error. I am not understanding why the generated H2O Pojo is not having required arguments in the function ...
0
votes
1answer
29 views

Is the best model always one with best test score, even though it looks overfit?

I'm making a binary classification model using gradient boosting (lightgbm). I usually use learning curves to check if my model is overfitting. The metric I'm using is sklearn's average precision-...
0
votes
0answers
10 views

GBDT- randomized repetition feature selection

Consider the following approach for feature selection in the specific case of gradient boosting decision trees: Randomly pick X% of features Run algorithm Record importance of each feature Repeat ...
0
votes
0answers
16 views

How to prevent GBDT from splitting on uninformative features?

I'm looking into using feature importance scores from GBDT for feature selection. Although GBDT does not need manual feature selection, the number of features is a restriction of the production system ...
0
votes
0answers
44 views

Regression Algorithms (Random Forests, GBM, GLMNET) Evaluation

Currently, I am doing a Machine Learning Project. I am in the process of performing evaluation of my models. As the title states, my models that I implemented were Random Forests, Gradient Boosting ...
1
vote
0answers
41 views

Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
3
votes
1answer
68 views

Did I understand AdaBoost correctly?

My mantra has always been that if you are not able to recreate something you haven't really understood it. In this manner I tried to implement the AdaBoost algorithm of Freund and Schapire I used one ...
4
votes
2answers
71 views

Machine Learning - Prediction Interval - Cheating?

I work at a company that is trying to use machine learning methods in particular gradient boosting and neural networks to make predictions on stock market data, so using historical data to predict ...
1
vote
1answer
44 views

Weak learners for XGBoost with Tweedie distribution

Could you please explain what are the standard weak learners for XGBoost when the objective parameter equals reg:tweedie? Are they GLMs (with Tweedie distribution of dependent variable) on all ...
0
votes
1answer
54 views

H2O GBM and Caret GBM

Hi I have doubt regarding the interaction. depth parameter in caret. I found a useful link hereabout interaction.depth in caret Now I am trying to find the similar parameter in H2O-GBM . Can anyone ...
0
votes
1answer
12 views

Using RF/GBM regressors when some observables are not real valued but just greater than a given value?

Say I can experimentally measure some number of $N$ data points which have an observable $y$ value I'm trying to model where $y=[0, 100)$ and is a continuously valued number. However, I also have $M$ ...
1
vote
1answer
76 views

Gradient boosting regression trained on skewed data

My target feature is right-skewed. I want to apply gradient boosting regression algorithm to predict it but I'm not sure what kind of preprocessing should I apply. As gradient boosting is based on ...
0
votes
0answers
12 views

Optimizing recursive loss functions with decision trees

For time series applications, it is often helpful to model things in a recursive fashion. For instance, let $f(x)$ be a model which predicts the next time step of some time series, so that $$ f(x^{n+...
1
vote
1answer
34 views

Does it make sense to use PCA right after GBM?

My Problem: I'm trying to classify a data into two groups as A and B based on 25 observations (data point) and 100 features. I used the Gradient Boosting Machine (GBM) to find out which feature has ...
2
votes
1answer
58 views

Gradient Boosted Trees vs Neural Network for limited data [closed]

I have a classification problem, with about 10 different inputs, some boolean, some categorical (and unrelated to each other), some being a float between 0 and 1, which need to be mapped to 4 ...
1
vote
0answers
24 views

Boosted decision trees: in which situations are “deep” decision trees performing better?

The general idea of boosted decision trees is to use very simple trees in the following manner (simplified, for intuition only): start with a simple tree, fit another simple tree on the residuals, ...
0
votes
0answers
8 views

GBM with an ordered y variable

I would like to execute a gbm in python, preferable in the h2o framework. My y variable is an ordered one, from to 1 to 5. I see that h2o does not have an implementation of an ordered logit. I ...
2
votes
2answers
44 views

May somebody help with interpretation of trees from h2o.gbm, see as photo attached

This picture is from h2o.gbm, while I'm not sure how to interpret the numbers in it. What is the big title "Class NO" mean? Does it mean the root node is labeled "No"? Or does it mean this tree is ...
0
votes
0answers
10 views

Differing feature importances after saving/loading GBT Classification model

I am using the GBTClassifier in Spark's ml.classification package. Right after training my model, I save it, and then I grabbed the feature importances, using ...
0
votes
0answers
22 views

Calculating log-likelihood or BIC of KNN and random forest/boosting models

I am trying to find the BIC of the KNN, random forest and boosting models (for regression, not classification) to use in a combined model that uses Bayesian model averaging to predict a target ...
0
votes
1answer
33 views

Boosting: In the function gbm() from library gbm and understanding the cv.folds implementation

In the function gbm() there is a parameter cv.folds. I know this is performing cross-validation but what I don't understand is how to interpret it. I tried extracting it using $cv.error but was ...
2
votes
1answer
34 views

Theoretically can gradient boosting achieve 100% of accuracy in an arbitrary dataset?

Consider gradient boosting like gbm or xgboost. I have a labelled dataset (X,y). If I don't care about over-fitting and I allow gbm or xgboost grow as much as needed, eventually can I reach the ...
10
votes
1answer
291 views

Boosting AND Bagging Trees (XGBoost, LightGBM)

There are many blog posts, YouTube videos, etc. about the ideas of bagging or boosting trees. My general understanding is that the pseudo code for each is: Bagging: Take N random samples of x% of ...
0
votes
0answers
57 views

Probability calibration from LightGBM model with class imbalance

I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. ...
0
votes
0answers
13 views

Why is the sqrt(n_features) the default maximum number of features for the best split in RandomForestClassifier? [duplicate]

Why does sklearn.ensemble.RandomForestClassifier references have $\sqrt{n}$ in the max_features implementation and why does randomForest in R seem to have the same $\sqrt{n}$ default? I am looking for ...
0
votes
0answers
10 views

Can I use Gradient Boosting Classier to determine feature importance without worrying about precision, recall and accuracy?

My boss is interested in understanding how certain actions improve user retention WoW. I decided to build a GBDT model to assess those features. My question is: Does accuracy, precision or recall ...
0
votes
0answers
26 views

Question about generalized boosted models (GBM) with Poisson assumption

I'm trying to learn how to use GBM R-package. I'm trying to model count data with the gradient boosting, so I'm using the Poisson distribution option in GBM. According to this PDF (page 12) http://...
0
votes
0answers
18 views

How can I run a decision tree algorithm with a specific hierarchy of variables and with many missing values?

I asked students in learning groups what their biggest learning problem was "today" for each learner. The biggest problem could either be "motivational" (=motivation problem) or cognitive (="knowledge ...
0
votes
0answers
15 views

Lars, glmboost & predict function in R

Below you can find a simple Monte Carlo simulation with 50 iterations. My goal is to print the mean-squared prediction error (MSPE) for lasso predictor and boosting with componentwise least squares. ...
0
votes
1answer
68 views

Interpreting RMSE of log-values

I am modelling a regression with a GBM and evaluate by RMSE. My model input & target is log-transformed which results in an RMSE that is also on log-scale. How can i interpret this in an ...
1
vote
1answer
80 views

Why the discrepancy between predict.xgb.Booster & xgboostexplainer prediction contributions?

One way to explain individual predictions of an xgb classifier is to calculate contributions of each feature. To my knowledge there are two packages in R that can do this for you automatically. In the ...
7
votes
2answers
1k views

Is it possible to combine predictions to improve overall prediction quality?

This is a binary classification problem. The metric that is being minimised is the log loss ( or cross entropy ). I also have an accuracy number, just for my information. It is a large, very ...
0
votes
0answers
18 views

What causes a high testing deviance vs. training deviance in a gradient boosting classifier?

My main goal is to classify multi-class data using supervised learning. Currently, I am looking into GradientBoostingClassifier as the estimator. I want to make sure I am selecting the model ...
1
vote
0answers
16 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
0
votes
1answer
54 views

focusing on hard examples in neural networks, like in gradient boosting?

gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar ...
0
votes
0answers
14 views

gradient with respect to totally different things in neural networks vs gradient boosting?

I'm confused about the usage of gradients in NN vs. GBM. Is is correct to say that the gradient is with respect to totally different things? my understanding is that: in NN (I'm following ...
0
votes
1answer
40 views

Cross-validation with Boosting Trees (do I need 4 sets?)

Normally, you have train, validation and test sets for training, tuning (hyperparameters) and finally evaluating a machine-learning model. If we use cross-validation, then we can effectively have only ...
0
votes
0answers
30 views

Counterfactual prediction with machine learning, sales data

I have a dataset from a supermarket with around 10 thousand products. The data has daily quantities and prices and discount information (whether the product had a discount and the size of the discount ...
1
vote
0answers
33 views

What is the relation between minimum instances per node and max depth?

In bagging and boosting models like random forest and xgboost we have hyper-parameters like minimum instances per node and max depth. If max depth is high the minimum instances per node will be less ...
1
vote
0answers
19 views

Boosted Regression Trees: Zero discrimination and calibration scores

I am using boosted regression trees using the gbm() and dismo() packages. When I run my models I get values of zero for discrimination and calibration in the $cv.statistics What would be the reason ...
0
votes
0answers
24 views

Scale response variable y in random forest or gradient boosted trees for regression == scale prediction?

Suppose we are fitting a random forest or gradient boosted tree model for regression on y. We first fit the model. Later on, we realized we need to fit y at another different scale, for example, a <...
0
votes
0answers
14 views

Results from BRTs and GAMMs differ

I have recently run some biological density data through a boosted regression trees (BRTs) and a generalised additive mixed-effects model (GAMM) to find the best environmental predictor of long-term ...
0
votes
0answers
22 views

how to learn a GradientBoostingClassifier model with with a fixed precision?

I would to have a final model with a fixed precision (say 0.75) and improve the recall as much as possible. How can i do that sklearn ?
1
vote
2answers
42 views

How NULLs in numerical variables are treated in tree-based models?

I understand that in tree-based models (CART, Gradient boosted trees, etc.), NULLs (i.e., NaN) in categorical variables can be treated as a separated category, while making node splits. However, how ...
1
vote
2answers
31 views

I am learning to do stacking. I want to know what will the input be to level 2 classifiers [closed]

In classification/regression problems, say if we use five different base classifiers, we get 5 predictions for each example. What would be the input to the second level classifiers?
0
votes
0answers
12 views

What a convex Precision-Recall curve means for training dataset?

Situation I have trained a GBDT model(gradient-boosted decision tree, a tree ensemble model) with a training dataset, and when I calculate PR curve on the same training set, it looks convex: For ...
0
votes
1answer
87 views

Is Gradient Boosting Regression Tree able to learn linear models

Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in [0,1]$. Assume one use gbdt to learn this linear model. And if ...