Questions tagged [adaboost]
A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.
131
questions
-2
votes
0
answers
57
views
Response and probability estimates in LogitBoost algortihm
I am reading the Additive logistic regression: a statistical view of boosting and from what I understand in boosting methods responses are usually encoded as [-1,1] instead of [0,1]. So the solution (...
2
votes
0
answers
38
views
Confusion about the code for choosing "stumps" in Adaboost algorithm
(I actually asked the following question on Stack Overflow recently:
https://stackoverflow.com/questions/76842431/confusion-about-the-code-for-choosing-stumps-in-adaboost-algorithm
but then I ...
- 121
2
votes
0
answers
66
views
AdaBoost vs GBM vs XGB [closed]
Can someone please summarise the difference between these 3 boosting algorithms: AdaBoost vs GBM vs XGB?
- 121
0
votes
0
answers
32
views
How does AdaBoost continue to work when its training error reaches zero?
It's said that continuing to boost after reaching zero training error can improve test error in AdaBoost.
However, I don't see how the algorithm can continue at all if it is reaching zero training ...
- 121
3
votes
1
answer
71
views
Does boosting being component-wise imply variable selection?
I have come across the term "component-wise" in the literature, and I am curious if this means that a model does perform variable selection. And not having this, mean it doesn't perform ...
- 33
1
vote
0
answers
61
views
Gradient Boosting interpretation of residuals
So I came across this sentence (from an e-learning website):
By training on the residuals of the model, this is an alternative means to give more importance to misclassified observations. Intuitively,...
- 155
1
vote
0
answers
20
views
Adaboost updating the weight. How to use Induction to proof two equation are the same?
I have these equation for updating the weight the same.
$$w^{(b+1)}_{i} = \frac{w_i^{(b)} \exp(-\alpha_by_ih_b(x_i))}{\sum_{j=1}^{n}w_j^{(b)} \exp(-\alpha_by_jh_b(x_j))}$$
and
$$w^{(b+1)}_{i}=\frac{\...
- 123
0
votes
0
answers
123
views
Why `adaboost` using `gbm` package will not produce exact binary prediction values such as `{1,0}`?
My input data contains a response variable is_disease which is obviously a Factor(Yes,...
- 329
1
vote
0
answers
63
views
Model which achieves good results on the Adult dataset, especially on the minority class
I'm looking for a model which achieves an high prediction and recall on the Adult dataset. I've tried with random forests, but couldn't get a model with good results on the minority class. Here are my ...
- 111
1
vote
1
answer
110
views
What’s the purpose of the 0.5 in the alpha term in Adaboost?
In the AdaBoost algorithm, the contribution of each weak learner is multiplied by a weight $\alpha$ given by
$$
\alpha_t = \frac{1}{2}ln\Big(\frac{1-\epsilon_t}{\epsilon_t}\Big)
$$
In the Elements of ...
- 483
1
vote
0
answers
39
views
Is there a stochastic AdaBoost?
In sklearn, the AdaBoostClassifier and AdaBoostRegressor classes do not have the subsample and max_features parameters, which are responsible for the stochastic approach to building a boosting model. ...
1
vote
1
answer
98
views
Boosting usa bootstraping?
I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
- 135
4
votes
2
answers
624
views
Can decision stumps have more than 2 leaves?
I understand decision stump: a shallow 1-level decision tree is often used as base-leaner in ensemble methods such as AdaBoost. What is not immediately clear to me ...
- 263
1
vote
1
answer
84
views
Divergence of accuracies and overfitting in AdaBoost
I implemented a binary classification setup of AdaBoost, but I train one model for each label in a one-vs-all arrangement, and in the prediction time I choose the class corresponding to the model with ...
- 131
1
vote
0
answers
44
views
Oscillation of AdaBoost Training error
Adaboost, using weak learners as Gaussian Naive bayes, has oscillating/unpredictable training error as we increase the number of weak learners. Is there a specific reason for this? Y-axis is the ...
- 29
4
votes
1
answer
919
views
Boosting reduces bias when compared to what algorithm?
I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
- 167
0
votes
1
answer
46
views
How to adjust sample weights in Adasboost
I am following this video tutorial to understand Adaboost
I am confused about the sample weights updating. It first calculates the amount of say of each stump by this formula,
where total error is ...
- 263
3
votes
2
answers
221
views
Where is Boosting applied in Gradient boosting techniques?
In boosting, the primary idea is to re-adjust weights of training instances, so that subsequent models learn how to fit difficult-to-classify samples.
From Wikipedia Boosting (Machine Learning):
...
- 657
1
vote
1
answer
69
views
how train AdaBoost M1 weak estimators?
I'm trying to implement AdaBoost.M1 as explained in Boosting: Foundations and Algorithms by Robert E. Schapire and Y. Freund. The problem is that I don't understand at each iteration t the estimator ...
- 131
0
votes
0
answers
56
views
Equation 10.2 from The Elements of Statistical Learning. Median of a chi-squared distribution
I'm reading about AdaBoost in the The Elements of Statistical Learning and I don't understand the equation 10.2. Below is an excerpt from the book.
The power of AdaBoost to dramatically increase the ...
- 45
0
votes
1
answer
317
views
Prediction of Adaboost Classifier between 0 and 1
I have to use an Adaboost classifier to predict if the data is a signal (1) or a background (0) event. But since the output is expected to be 1 or 0 the ...
- 57
2
votes
0
answers
153
views
Generalisation error bound proof for the Adaboost algorithm, corollary 4.4 in Freund and Schapire (2014)
I am having difficulty with a detail in the proof of generalisation error bounds for the AdaBoost algorithm, and would appreciate some assistance. This is the proof of corollary 4.4 on page 54 of ...
- 2,425
5
votes
1
answer
485
views
Why does classifier (XGBoost) "after PCA" runtime increase compared to "before PCA"
The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
- 51
0
votes
0
answers
282
views
How does resampling in AdaBoost (exactly) work?
Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
- 157
1
vote
1
answer
2k
views
What's the purpose of learning rate in sklearn AdaBoost implementation
We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner.
Conceptually, ...
- 263
1
vote
0
answers
112
views
Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?
I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
1
vote
0
answers
126
views
Interpretation of a tightly paired learning curve with increasing loss
I am assessing models for a binary classification task and have created a model with a very strange learning curve. This is the learning curve of an sklearn AdaBoostClassifier fitted with default ...
- 11
0
votes
1
answer
218
views
Why is AdaBoost with short decision trees a form of feature selection?
Why is AdaBoost with short decision trees a form of feature selection? What is so special with short decision trees?
6
votes
2
answers
3k
views
AdaBoost - why decision stumps instead of trees?
Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
- 229
1
vote
1
answer
389
views
Why does AdaBoost algorithm use weighted data points?
I am learning about AdaBoost algorithm. At each iteration, adaBoost set higher weight to mistake datapoints, and lower weight to correct classified data points. I do not understand why the algorithm ...
- 557
1
vote
0
answers
28
views
what will be modified algorithm if this loss function will be modified? [closed]
I have a loss function of AdaBoost and $l(h(x),y)=e^{(-y*h(x))}$ and what will be modified algorithm if we modify loss function to $l(h(x),y)=ln(1+e^{(-y*h(x)})$.
I was not able to find a way to ...
- 51
2
votes
2
answers
911
views
Does Adaboost ensemble use bootstrapping?
I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as
I understand the general idea behind it: Give more weight to ...
0
votes
1
answer
250
views
How to generate a confidence interval for an adaboost prediction?
I have created a simple AdaBoostRegressor (sklearn) model which is trained on a feature set $X$ (house features) to predict a variable $y$ (house prices). The model can be used to create a prediction ...
- 23
1
vote
0
answers
192
views
What is a reasonable number of splits (maximum) for a ensemble classifier?
I am trying to use an ensemble classifier (honing in on Matlab fitcensemble). I've also explored using a single decision tree as well as tree bagging (Matlab fitctree, TreeBagger)
Simple binary (A/B) ...
- 51
3
votes
1
answer
184
views
in which part is used the splitting criteria in AdaBoost?
I have been reading the original article about AdaBoost and by comparing with other reading material it has come some doubts about this model. Please feel free to correct me if in any part I am wrong. ...
- 601
0
votes
0
answers
26
views
Getting bad results when testing my prediction model on new patients
I analyze medical diagnostic labeled data: the independent variables are medical parameters (assume that one of the variables is blood pressure) and the dependent variable is 1 or 0. The data contains ...
- 105
0
votes
0
answers
224
views
What is this symbol in Adaboost algorithm (SAMME)?
According to the original paper of SAMME algorithm (Adaboost for multiclass problems), it is described as follows:
The general idea is very straightfoward, except for this symbol:
The author didn't ...
- 133
2
votes
2
answers
1k
views
Improve Adaboost that using weighted logistic regression instead of decision trees
I implemented Adaboost that using weighted logistic regression instead of decision trees and I managed to get to 0.5% error, I'm trying to improve it for days with no success and I know it possible to ...
- 21
6
votes
0
answers
110
views
Weighted Conditional Expectation definition in AdaBoost
I am looking at "Additive logistic regression a statistical view of boosting" paper (https://web.stanford.edu/~hastie/Papers/AdditiveLogisticRegression/alr.pdf)
In page 346, the authors ...
- 61
1
vote
2
answers
666
views
Adaboost -- how does reweighting affect the learning process for the subsequent learner?
In Adaboost, when you reweight the samples, how does the training process for the next classifier in the boosting algorithm take in to account the weights? Is it reflected in the loss function of the ...
- 414
3
votes
2
answers
715
views
Random forest after cross validation
i have been wondering for some time now how random forests (or AdaBoost, doesn't matter) are built when using cross-validation.
Let's see we're using 5-fold cross validation to train random forests on ...
1
vote
1
answer
463
views
Weights in Adaboost
The Adaboost algorithm is shown below.
I have a couple of questions.
First, I am a little confused on how the weights for each training sample is applied when using Adaboost. From looking at the ...
- 605
1
vote
1
answer
520
views
AdaBoost assumption of weak classifier
I've read and think I got a good grasp of the math behind AdaBoost, but I wasn't able to understand why AdaBoost requires a weak base-classifier?
Specifically, I'm dealing with AdaBoost using ...
- 113
1
vote
0
answers
34
views
Creating Predictions From Base Regressor Models in a Boosted Machine
So I'm writing my own implementation of AdaBoost using chapter 10 of Elements of Statistical Learning. The pseudocode given in the book lists how to make the algorithm for a classification problem, ...
- 297
1
vote
0
answers
148
views
AdaBoost Weight Updating for Misclassified Points
http://www.inf.fu-berlin.de/inst/ag-ki/adaboost4.pdf
I was reading this paper and came across the question showed in the second image. However, in the first image provided in a book on Machine ...
- 11
0
votes
0
answers
492
views
Why does AdaBoost use decision stumps instead of 0-depth trees?
Why is it that AdaBoost uses decision stumps for the weak learners? It seems simpler to me to just use the weighted majority of the data points for the classification. Why shouldn't we do this?
- 1
0
votes
0
answers
330
views
How to update the weight when Adaboost is used for regression
When Adaboost is used for regression, in the $m$ step, we suppose $G_m(x)$ is $m$-classifier with sample $\{(x_i,y_i)\}$; the error of i-th sample is $e^i_m;$ the weight of $e^i_m$ is $w^i_m;$ the ...
- 1,245
0
votes
0
answers
241
views
Termination Condition for AdaBoost.R2
I can't quite wrap my head around the termination condition of AdaBoost.R2 as defined by Drucker in this paper. On page 2 of the paper he states to "repeat the following while the average loss* $\bar{...
- 1
1
vote
1
answer
165
views
What is minimized/optimized when we use AdaBoost
When I learned about CART, we learned that at each split, we try to minimize some measure (usually Gini index) of the split. That is, we determine the predictor and threshold that decreases the Gini ...
- 3,063
4
votes
1
answer
5k
views
How do you interpret prediction output in GBM() in R for classification problem?
I created a model using the gbm() function in library(gbm). Within the gbm() function, I set the distribution as "adaboost". I have a binary response [0, 1]. I used the predict.gbm function for ...
- 3,063