Boosting is a process of finding & combining weakly predictive models into a strongly predictive model.

learn more… | top users | synonyms

0
votes
0answers
23 views

Methods for supervised regression learning

What ML methods should I try for a data set of around 1000 samples? The output variable is dependent on say 10 regressors of which 8 are real numbers and the other 2 are categorical. Or they might ...
2
votes
0answers
35 views

Boosting: why is the learning rate considered a regularization parameter?

I understand that the learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically small trees- that is added in the series. It was shown ...
0
votes
0answers
20 views

Error in gbm.step: In cor(y_i, u_i) : the standard deviation is zero [closed]

My data is called: data(Normalised.scores) I am attempting to construct a boosted regression tree using the function gbm.step() using the dismo package, where ...
0
votes
0answers
16 views

error family in boosted regression tree: gbm package

I am trying to understand boosted regression tree. I am using the gbm package in R. I noticed that in this package one has to specify the error family. I understand ...
1
vote
0answers
11 views

What is the best achieved/possible score for KDD 2009 with Boosted Decision Trees? [closed]

Could you please let me know what is the best achieved or possible score for KDD 2009 churn dataset with Boosted Decision Trees?
0
votes
0answers
15 views

derive loss function for gamma regression

In the R package mboost there is a family called "GammaReg" which implementes "negative Gamma log-likelihood with logarithmic link function". Still, I don't really ...
1
vote
1answer
67 views

Intuitive explanations of differences between Gradient Boosting Trees (GBM) & Adaboost

I'm trying to understand the differences between GBM & Adaboost. These are what I've understood so far: There are both boosting algorithms, which learns from previous model's errors and ...
1
vote
1answer
38 views

How many AdaBoost iterations?

In one R package, ada, the main AdaBoost fitting function (also called ada) takes an argument specifying the number of boosting ...
1
vote
1answer
44 views

The relationship between adaboost and gradient boosting

I am reading the chapter 10 of "The Elements of Statistical Learning 2nd ed, (ESLII)", where the Adaboost algorithm is explained by minimizing the exponential loss using stagewise additive modelling ...
2
votes
0answers
46 views

Modelling clustered data using boosted regression trees

I'm modelling habitat selection using boosted regression trees (BRTs), which I prefer over linear models for a variety of reasons (modeling complex nonlinear relationships and interactions, ...
3
votes
1answer
34 views

Adaboost - update of weights

i am self-studying AdaBoost - and reading the following useful article. http://www.inf.fu-berlin.de/inst/ag-ki/adaboost4.pdf . I am trying to understand, as per below, the following questions: 1) ...
4
votes
1answer
125 views

Relative variable importance for Boosting

I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like: The measures are based on the number of times a ...
2
votes
1answer
58 views

A question about Dynamic Random Forest

On this article, Simon Bernard proposes a new approach for constructing Random Forest called Dynamic Random Forest. I am new on this subject, so after reading the article, I have a doubt regarding the ...
6
votes
4answers
1k views

Is automated machine learning a dream?

As I discover machine learning I see different interesting techniques such as: automatically tune algorithms with techniques such as grid search, get more ...
0
votes
0answers
11 views

Consistency of values of coefficients ($\alpha$) in Adaboost

Based on the given set of data, Adaboost learns weights($\alpha$) corresponding to the weak classifiers. Now suppose that there is a huge chunk of data broken down into smaller parts, e.g: a total of ...
11
votes
3answers
226 views

When should I not use an ensemble classifier?

In general, in a classification problem where the goal is to accurately predict out-of-sample class membership, when should I not to use an ensemble classifier? This question is closely related to ...
2
votes
1answer
113 views

Scikit Binomial Deviance Loss Function

This is scikit GradientBoosting's binomial deviance loss function, ...
3
votes
1answer
93 views

How to use decision stump as weak learner in Adaboost?

I want to implement Adaboost using Decision Stump. Is it correct to make as many decision stump as our data set's features in each iteration of Adaboost? For example, if I have a data set with 24 ...
3
votes
2answers
80 views

When does it makes sense to use Cross Validation?

My understanding is that cross validation is about using different chunks of the training data to train the model and average out the error estimation so that the variance is less. For example, in ...
0
votes
0answers
17 views

How many iterations we have to perform in adaboost classification?

How many iterations we have to perform in adaboost classification? As the number of iteration increases error rate gradually reduces and sometimes classification accuracy goes upto 100% in both ...
5
votes
1answer
138 views

Where must we use Bagging or Boosting?

I want to know when Bagging is better than Boosting? How I select appropriate method for my classification task? I think when we have many outliers in our data-set, Bagging must be better than ...
0
votes
1answer
421 views

Reconciling boosted regression trees (BRT), generalized boosted models (GBM), and gradient boosting machine (GBM)

Questions: What is the difference(s) between boosted regression trees (BRT) and generalized boosted models (GBM)? Can they be used interchangeably? Is one a specific form of the other? Why did ...
3
votes
1answer
111 views

Random forest vs Adaboost

In section 7 of the paper Random Forests (Breiman, 1999), the author states the following conjecture: "Adaboost is a Random Forest". Has anyone proved, or disproved this? What has been done to prove ...
1
vote
0answers
277 views

Xgboost Regresion tree

I am building a boosted regression tree in R and I use the simple xgboost function from the package xgboost in R. ...
2
votes
1answer
67 views

Meaning of the Boosting algorithm for Regression Trees

I have a problem with understanding the concept of the Boosting Algorithm. ...
1
vote
1answer
129 views

out-of-bag error estimate for Boosting?

In Random Forest, each tree is grown in parallel on a unique boostrap sample of the data. Because each boostrap sample is expected to contain about 63% of unique observations, this lefts roughly 37% ...
0
votes
0answers
71 views

Huge overfitting with Random Forests and Boosted Trees?

In the following picture, the boxplots represent a performance metric (the closer to 1, the better) recorded for 50 runs of cross-validation, and the black filled circles are the training values of ...
3
votes
2answers
176 views

Are Random Forests and Boosting parametric or non-parametric?

From this excellent paper by Breiman, we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forests, ...
1
vote
0answers
85 views

Boosting Explained

I'm a newbie trying to learn Boosting. The examples I found online are quite confusing. Is there a simple tutorial somewhere that explains ...
3
votes
0answers
33 views

GradientBoostClassifier(sklearn) takes very long time to train

I'm using dataset with 61879 datapoints and 102 features. On this dataset Randomforest(sklearn) takes less than 90s to train for 100 estimators while GradientBoostClassifier(sklearn) is taking forever ...
1
vote
1answer
61 views

Skewed Classification Problem

So I've read around and seen this is a problem. I have a classification problem and 12 variables ... I'm working on getting more, but even if l get the number to 20-30 I feel like the problem will ...
0
votes
1answer
121 views

Meaning of `max_depth` in GradientBoostingClassifier in scikit-learn

when I use the GradientBoostingClassifier from scikit-learn, I find that there is a parameter max_depth to set, which controls the maximum depth of the regression ...
1
vote
1answer
40 views

How to choose a regression tree (base learner) at each iteration of Gradient Tree Boosting?

I'm trying to understand Gradient Tree Boosting, by following Prof. Friedman's original paper: Greedy Function Approximation: A Gradient Boosting Machine. Basically, at each iteration, a regression ...
1
vote
1answer
29 views

Why does the equation $ -\sum^{n}_{t=1} \tilde{W}(t)_{m-1} y_{t}h(x; \theta_{m}) = 2 \epsilon_m -1$ hold in boosting?

I was trying to understand the boosting algorithm as described by the MIT graduate class lectures notes on ocw. On page 2 they give the outline of boosting as follows: The step that is not clear ...
1
vote
2answers
166 views

Boosted trees and Variable Interactions

How can one see in a Boosted trees classification model, which variables interact with each other and how much? I would like to make use o this in R gbm package if possible
0
votes
0answers
72 views

Value of the loss function and Classification Errors in gbm package (R)

I have a simple problem of classification (0s and 1s) using adaboost loss function. When I check the components of a boosted model using the gbm package I see: ...
0
votes
0answers
57 views

why boosting method is sensitive to outliers

I found many articles showing that boosting methods are sensitive to outliers, but no article explains why. In my experience, I feel outliers data is bad for any machine learning algorithms, but why ...
1
vote
1answer
43 views

Alternative to AIC for feature selection in classification

I want to know what are the most common methods for feature selection in classification problems (binary and mutli-class). I see in Chapter 6 of Zumel and Mount that they use AIC before they train ...
0
votes
0answers
70 views

what is the difference between bagging and boosting in random forest?

I understand what is bagging and how it is applied to random forest. But how is bagging different from boosting. If boosting is different from bagging, how can boosting be applied to random forest?
3
votes
1answer
88 views

How are individual trees added together in boosted regression tree?

I'm reading Introduction to Statistical Learning, James, G., et al. (2013), in which they describe the Boosted Regression Tree algorithm as following. What I do not understand is Eq 8.10 and 8.11. ...
0
votes
0answers
119 views

Best machine learning methtod for classificating datasets with non-independent cases within the groups

I have to perform binary classification of my data with supervised machine learning, but I have some difficulties working with my data set. It consists many genetic mutations that have parameters ...
1
vote
1answer
101 views

AdaBoost over blackbox weak classifier

Can I somehow implement AdaBoost procedure over a weak classifier from another library? For example over SVM from libsvm, or over some neural network. The idea of AdaBoost is that current weights of ...
13
votes
1answer
199 views

Prediction interval based on cross-validation (CV)

In the text books and youtube lectures I learned a lot about iterative models such as boosting, but I never saw anything about deriving a prediction interval. Cross validation is used for the ...
0
votes
0answers
132 views

Combining multiple feature subsets through ensemble classification methods?

I have a set of $N$ samples to be classifies in a binary classification problem. I have extracted features from these samples from 4 different perspectives (views) of every samples. Hence I have 4 ...
1
vote
0answers
51 views

What is “fitted function” in the context of boosted regression tree?

I'm following the tutorial of package dismo's boosted regression tree, which produces two graphs, about fitted function and ...
1
vote
2answers
29 views

Incremental improvement for boosting

By adding additional factors, will the fitting result of a boosting algo (say Ada boosting) guaranteed to be improved? From my experiment, adding additional factors could make the prediction accuracy ...
1
vote
1answer
159 views

Posterior probabilities with decision trees or decision forests

Is there a way to get posterior probabilities $P(C | \vec{x})$ (probability that a data item $\vec{x}$ belong to one of the given classes) in a multiclass classification problem using decision trees ...
0
votes
2answers
26 views

For the given type of dataset, what would generally be the set of classifiers that should be tried to get the highest TPR for FPR = 0.01

I'm primarily looking to attain the maximum True Positive Rate for a small False positive Rate of say 0.01. The following is an instance: 1, 37.33, 228.39, 0, 77.060599, 0.073384, 0.052536, 1.389826, ...
2
votes
2answers
741 views

Why Adaboost with Decision Trees?

I've been reading a bit on boosting algorithms for classification tasks and Adaboost in particular. I understand that the purpose of Adaboost is to take several "weak learners" and, through a set of ...
0
votes
1answer
222 views

How to choose an appropriate maxdepth in rpart.conrol?

I'm using the boosting method in adabag library and trying to choose an appropriate maxdepth in rpart.control for building a 2-class classification model using my training dataset. I have noticed that ...