Questions tagged [adaboost]

A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
57 views

Response and probability estimates in LogitBoost algortihm

I am reading the Additive logistic regression: a statistical view of boosting and from what I understand in boosting methods responses are usually encoded as [-1,1] instead of [0,1]. So the solution (...
useron3wrnol's user avatar
2 votes
0 answers
38 views

Confusion about the code for choosing "stumps" in Adaboost algorithm

(I actually asked the following question on Stack Overflow recently: https://stackoverflow.com/questions/76842431/confusion-about-the-code-for-choosing-stumps-in-adaboost-algorithm but then I ...
Richard's user avatar
  • 121
2 votes
0 answers
66 views

AdaBoost vs GBM vs XGB [closed]

Can someone please summarise the difference between these 3 boosting algorithms: AdaBoost vs GBM vs XGB?
Amina Umar's user avatar
0 votes
0 answers
32 views

How does AdaBoost continue to work when its training error reaches zero?

It's said that continuing to boost after reaching zero training error can improve test error in AdaBoost. However, I don't see how the algorithm can continue at all if it is reaching zero training ...
HappyFace's user avatar
  • 121
3 votes
1 answer
71 views

Does boosting being component-wise imply variable selection?

I have come across the term "component-wise" in the literature, and I am curious if this means that a model does perform variable selection. And not having this, mean it doesn't perform ...
paulmuaddib's user avatar
1 vote
0 answers
61 views

Gradient Boosting interpretation of residuals

So I came across this sentence (from an e-learning website): By training on the residuals of the model, this is an alternative means to give more importance to misclassified observations. Intuitively,...
Caterina's user avatar
  • 155
1 vote
0 answers
20 views

Adaboost updating the weight. How to use Induction to proof two equation are the same?

I have these equation for updating the weight the same. $$w^{(b+1)}_{i} = \frac{w_i^{(b)} \exp(-\alpha_by_ih_b(x_i))}{\sum_{j=1}^{n}w_j^{(b)} \exp(-\alpha_by_jh_b(x_j))}$$ and $$w^{(b+1)}_{i}=\frac{\...
Kimbly  North's user avatar
0 votes
0 answers
123 views

Why `adaboost` using `gbm` package will not produce exact binary prediction values such as `{1,0}`?

My input data contains a response variable is_disease which is obviously a Factor(Yes,...
nobodyishere's user avatar
1 vote
0 answers
63 views

Model which achieves good results on the Adult dataset, especially on the minority class

I'm looking for a model which achieves an high prediction and recall on the Adult dataset. I've tried with random forests, but couldn't get a model with good results on the minority class. Here are my ...
Gerardo Zinno's user avatar
1 vote
1 answer
110 views

What’s the purpose of the 0.5 in the alpha term in Adaboost?

In the AdaBoost algorithm, the contribution of each weak learner is multiplied by a weight $\alpha$ given by $$ \alpha_t = \frac{1}{2}ln\Big(\frac{1-\epsilon_t}{\epsilon_t}\Big) $$ In the Elements of ...
user1809989's user avatar
1 vote
0 answers
39 views

Is there a stochastic AdaBoost?

In sklearn, the AdaBoostClassifier and AdaBoostRegressor classes do not have the subsample and max_features parameters, which are responsible for the stochastic approach to building a boosting model. ...
Владислав Гаджиханов's user avatar
1 vote
1 answer
98 views

Boosting usa bootstraping?

I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
PicaPython's user avatar
4 votes
2 answers
624 views

Can decision stumps have more than 2 leaves?

I understand decision stump: a shallow 1-level decision tree is often used as base-leaner in ensemble methods such as AdaBoost. What is not immediately clear to me ...
arilwan's user avatar
  • 263
1 vote
1 answer
84 views

Divergence of accuracies and overfitting in AdaBoost

I implemented a binary classification setup of AdaBoost, but I train one model for each label in a one-vs-all arrangement, and in the prediction time I choose the class corresponding to the model with ...
Mehdi Abbassi's user avatar
1 vote
0 answers
44 views

Oscillation of AdaBoost Training error

Adaboost, using weak learners as Gaussian Naive bayes, has oscillating/unpredictable training error as we increase the number of weak learners. Is there a specific reason for this? Y-axis is the ...
A J's user avatar
  • 29
4 votes
1 answer
919 views

Boosting reduces bias when compared to what algorithm?

I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
Slim Shady's user avatar
0 votes
1 answer
46 views

How to adjust sample weights in Adasboost

I am following this video tutorial to understand Adaboost I am confused about the sample weights updating. It first calculates the amount of say of each stump by this formula, where total error is ...
ycenycute's user avatar
  • 263
3 votes
2 answers
221 views

Where is Boosting applied in Gradient boosting techniques?

In boosting, the primary idea is to re-adjust weights of training instances, so that subsequent models learn how to fit difficult-to-classify samples. From Wikipedia Boosting (Machine Learning): ...
The Wanderer's user avatar
1 vote
1 answer
69 views

how train AdaBoost M1 weak estimators?

I'm trying to implement AdaBoost.M1 as explained in Boosting: Foundations and Algorithms by Robert E. Schapire and Y. Freund. The problem is that I don't understand at each iteration t the estimator ...
Mehdi Abbassi's user avatar
0 votes
0 answers
56 views

Equation 10.2 from The Elements of Statistical Learning. Median of a chi-squared distribution

I'm reading about AdaBoost in the The Elements of Statistical Learning and I don't understand the equation 10.2. Below is an excerpt from the book. The power of AdaBoost to dramatically increase the ...
Jedrek369's user avatar
0 votes
1 answer
317 views

Prediction of Adaboost Classifier between 0 and 1

I have to use an Adaboost classifier to predict if the data is a signal (1) or a background (0) event. But since the output is expected to be 1 or 0 the ...
Wolfmercury's user avatar
2 votes
0 answers
153 views

Generalisation error bound proof for the Adaboost algorithm, corollary 4.4 in Freund and Schapire (2014)

I am having difficulty with a detail in the proof of generalisation error bounds for the AdaBoost algorithm, and would appreciate some assistance. This is the proof of corollary 4.4 on page 54 of ...
microhaus's user avatar
  • 2,425
5 votes
1 answer
485 views

Why does classifier (XGBoost) "after PCA" runtime increase compared to "before PCA"

The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
appeldaniel's user avatar
0 votes
0 answers
282 views

How does resampling in AdaBoost (exactly) work?

Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
Christian's user avatar
  • 157
1 vote
1 answer
2k views

What's the purpose of learning rate in sklearn AdaBoost implementation

We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner. Conceptually, ...
arilwan's user avatar
  • 263
1 vote
0 answers
112 views

Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?

I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
supratim saha's user avatar
1 vote
0 answers
126 views

Interpretation of a tightly paired learning curve with increasing loss

I am assessing models for a binary classification task and have created a model with a very strange learning curve. This is the learning curve of an sklearn AdaBoostClassifier fitted with default ...
callum's user avatar
  • 11
0 votes
1 answer
218 views

Why is AdaBoost with short decision trees a form of feature selection?

Why is AdaBoost with short decision trees a form of feature selection? What is so special with short decision trees?
Agus Bendoro's user avatar
6 votes
2 answers
3k views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
qalis's user avatar
  • 229
1 vote
1 answer
389 views

Why does AdaBoost algorithm use weighted data points?

I am learning about AdaBoost algorithm. At each iteration, adaBoost set higher weight to mistake datapoints, and lower weight to correct classified data points. I do not understand why the algorithm ...
i_love_somebody's user avatar
1 vote
0 answers
28 views

what will be modified algorithm if this loss function will be modified? [closed]

I have a loss function of AdaBoost and $l(h(x),y)=e^{(-y*h(x))}$ and what will be modified algorithm if we modify loss function to $l(h(x),y)=ln(1+e^{(-y*h(x)})$. I was not able to find a way to ...
jack vorobey's user avatar
2 votes
2 answers
911 views

Does Adaboost ensemble use bootstrapping?

I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as I understand the general idea behind it: Give more weight to ...
Alberto Perez Martinez's user avatar
0 votes
1 answer
250 views

How to generate a confidence interval for an adaboost prediction?

I have created a simple AdaBoostRegressor (sklearn) model which is trained on a feature set $X$ (house features) to predict a variable $y$ (house prices). The model can be used to create a prediction ...
James T's user avatar
  • 23
1 vote
0 answers
192 views

What is a reasonable number of splits (maximum) for a ensemble classifier?

I am trying to use an ensemble classifier (honing in on Matlab fitcensemble). I've also explored using a single decision tree as well as tree bagging (Matlab fitctree, TreeBagger) Simple binary (A/B) ...
bliswell's user avatar
3 votes
1 answer
184 views

in which part is used the splitting criteria in AdaBoost?

I have been reading the original article about AdaBoost and by comparing with other reading material it has come some doubts about this model. Please feel free to correct me if in any part I am wrong. ...
Layla's user avatar
  • 601
0 votes
0 answers
26 views

Getting bad results when testing my prediction model on new patients

I analyze medical diagnostic labeled data: the independent variables are medical parameters (assume that one of the variables is blood pressure) and the dependent variable is 1 or 0. The data contains ...
Amit S's user avatar
  • 105
0 votes
0 answers
224 views

What is this symbol in Adaboost algorithm (SAMME)?

According to the original paper of SAMME algorithm (Adaboost for multiclass problems), it is described as follows: The general idea is very straightfoward, except for this symbol: The author didn't ...
joann2555's user avatar
  • 133
2 votes
2 answers
1k views

Improve Adaboost that using weighted logistic regression instead of decision trees

I implemented Adaboost that using weighted logistic regression instead of decision trees and I managed to get to 0.5% error, I'm trying to improve it for days with no success and I know it possible to ...
Oran Sherf's user avatar
6 votes
0 answers
110 views

Weighted Conditional Expectation definition in AdaBoost

I am looking at "Additive logistic regression a statistical view of boosting" paper (https://web.stanford.edu/~hastie/Papers/AdditiveLogisticRegression/alr.pdf) In page 346, the authors ...
Aggamarcel's user avatar
1 vote
2 answers
666 views

Adaboost -- how does reweighting affect the learning process for the subsequent learner?

In Adaboost, when you reweight the samples, how does the training process for the next classifier in the boosting algorithm take in to account the weights? Is it reflected in the loss function of the ...
student010101's user avatar
3 votes
2 answers
715 views

Random forest after cross validation

i have been wondering for some time now how random forests (or AdaBoost, doesn't matter) are built when using cross-validation. Let's see we're using 5-fold cross validation to train random forests on ...
Steven Niggebrugge's user avatar
1 vote
1 answer
463 views

Weights in Adaboost

The Adaboost algorithm is shown below. I have a couple of questions. First, I am a little confused on how the weights for each training sample is applied when using Adaboost. From looking at the ...
David's user avatar
  • 605
1 vote
1 answer
520 views

AdaBoost assumption of weak classifier

I've read and think I got a good grasp of the math behind AdaBoost, but I wasn't able to understand why AdaBoost requires a weak base-classifier? Specifically, I'm dealing with AdaBoost using ...
Ruslan's user avatar
  • 113
1 vote
0 answers
34 views

Creating Predictions From Base Regressor Models in a Boosted Machine

So I'm writing my own implementation of AdaBoost using chapter 10 of Elements of Statistical Learning. The pseudocode given in the book lists how to make the algorithm for a classification problem, ...
Jonathan Bechtel's user avatar
1 vote
0 answers
148 views

AdaBoost Weight Updating for Misclassified Points

http://www.inf.fu-berlin.de/inst/ag-ki/adaboost4.pdf I was reading this paper and came across the question showed in the second image. However, in the first image provided in a book on Machine ...
pineapple69's user avatar
0 votes
0 answers
492 views

Why does AdaBoost use decision stumps instead of 0-depth trees?

Why is it that AdaBoost uses decision stumps for the weak learners? It seems simpler to me to just use the weighted majority of the data points for the classification. Why shouldn't we do this?
MLNewbie's user avatar
0 votes
0 answers
330 views

How to update the weight when Adaboost is used for regression

When Adaboost is used for regression, in the $m$ step, we suppose $G_m(x)$ is $m$-classifier with sample $\{(x_i,y_i)\}$; the error of i-th sample is $e^i_m;$ the weight of $e^i_m$ is $w^i_m;$ the ...
user6703592's user avatar
  • 1,245
0 votes
0 answers
241 views

Termination Condition for AdaBoost.R2

I can't quite wrap my head around the termination condition of AdaBoost.R2 as defined by Drucker in this paper. On page 2 of the paper he states to "repeat the following while the average loss* $\bar{...
Sigmaeon's user avatar
1 vote
1 answer
165 views

What is minimized/optimized when we use AdaBoost

When I learned about CART, we learned that at each split, we try to minimize some measure (usually Gini index) of the split. That is, we determine the predictor and threshold that decreases the Gini ...
confused's user avatar
  • 3,063
4 votes
1 answer
5k views

How do you interpret prediction output in GBM() in R for classification problem?

I created a model using the gbm() function in library(gbm). Within the gbm() function, I set the distribution as "adaboost". I have a binary response [0, 1]. I used the predict.gbm function for ...
confused's user avatar
  • 3,063