Questions tagged [adaboost]
A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.
122
questions
1
vote
1
answer
22
views
Boosting usa bootstraping?
I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
- 125
4
votes
2
answers
231
views
Can decision stumps have more than 2 leaves?
I understand decision stump: a shallow 1-level decision tree is often used as base-leaner in ensemble methods such as AdaBoost. What is not immediately clear to me ...
- 231
1
vote
1
answer
25
views
Divergence of accuracies and overfitting in AdaBoost
I implemented a binary classification setup of AdaBoost, but I train one model for each label in a one-vs-all arrangement, and in the prediction time I choose the class corresponding to the model with ...
- 137
0
votes
0
answers
19
views
How AdaBoost will treat weak learners with certain errors?
I have to answer the following question and Im struggling:
For each of the following cases, explain how AdaBoost will treat the weak learner $G_k$ with the weighted error $\text{err}_k(G_k, w_k) = \...
- 65
1
vote
0
answers
13
views
Oscillation of AdaBoost Training error
Adaboost, using weak learners as Gaussian Naive bayes, has oscillating/unpredictable training error as we increase the number of weak learners. Is there a specific reason for this? Y-axis is the ...
- 11
2
votes
1
answer
93
views
Boosting reduces bias when compared to what algorithm?
I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
- 203
0
votes
1
answer
20
views
How to adjust sample weights in Adasboost
I am following this video tutorial to understand Adaboost
I am confused about the sample weights updating. It first calculates the amount of say of each stump by this formula,
where total error is ...
- 233
0
votes
0
answers
83
views
Confusion with AdaBoost.M1 algorithm
I'm having my head buried in the AdaBoost.M1. Are there yet some variations of AdaBoost.M1? I ask this because I read variations ...
- 231
0
votes
0
answers
21
views
What is the use of the learning_rate parameter in sklearn AdaBoostClassifier/Regressor?
There is a similar question here. But I have doubts regarding the answer. Hence am asking a new question clarifying my doubt.
The general pattern for boosting is that the predictions of the weak ...
3
votes
2
answers
112
views
Where is Boosting applied in Gradient boosting techniques?
In boosting, the primary idea is to re-adjust weights of training instances, so that subsequent models learn how to fit difficult-to-classify samples.
From Wikipedia Boosting (Machine Learning):
...
- 647
1
vote
1
answer
28
views
how train AdaBoost M1 weak estimators?
I'm trying to implement AdaBoost.M1 as explained in Boosting: Foundations and Algorithms by Robert E. Schapire and Y. Freund. The problem is that I don't understand at each iteration t the estimator ...
- 137
0
votes
0
answers
45
views
Equation 10.2 from The Elements of Statistical Learning. Median of a chi-squared distribution
I'm reading about AdaBoost in the The Elements of Statistical Learning and I don't understand the equation 10.2. Below is an excerpt from the book.
The power of AdaBoost to dramatically increase the ...
- 23
0
votes
1
answer
60
views
Prediction of Adaboost Classifier between 0 and 1
I have to use an Adaboost classifier to predict if the data is a signal (1) or a background (0) event. But since the output is expected to be 1 or 0 the ...
- 57
2
votes
0
answers
65
views
Generalisation error bound proof for the Adaboost algorithm, corollary 4.4 in Freund and Schapire (2014)
I am having difficulty with a detail in the proof of generalisation error bounds for the AdaBoost algorithm, and would appreciate some assistance. This is the proof of corollary 4.4 on page 54 of ...
- 2,206
3
votes
0
answers
165
views
Why does classifier (XGBoost) "after PCA" runtime increase compared to "before PCA"
The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
- 31
0
votes
0
answers
132
views
How does resampling in AdaBoost (exactly) work?
Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
- 125
1
vote
1
answer
667
views
What's the purpose of learning rate in sklearn AdaBoost implementation
We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner.
Conceptually, ...
- 231
1
vote
0
answers
57
views
Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?
I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
1
vote
0
answers
45
views
Interpretation of a tightly paired learning curve with increasing loss
I am assessing models for a binary classification task and have created a model with a very strange learning curve. This is the learning curve of an sklearn AdaBoostClassifier fitted with default ...
- 11
0
votes
1
answer
69
views
Why is AdaBoost with short decision trees a form of feature selection?
Why is AdaBoost with short decision trees a form of feature selection? What is so special with short decision trees?
2
votes
2
answers
1k
views
AdaBoost - why decision stumps instead of trees?
Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
- 179
1
vote
1
answer
79
views
Why does AdaBoost algorithm use weighted data points?
I am learning about AdaBoost algorithm. At each iteration, adaBoost set higher weight to mistake datapoints, and lower weight to correct classified data points. I do not understand why the algorithm ...
- 557
1
vote
0
answers
23
views
what will be modified algorithm if this loss function will be modified? [closed]
I have a loss function of AdaBoost and $l(h(x),y)=e^{(-y*h(x))}$ and what will be modified algorithm if we modify loss function to $l(h(x),y)=ln(1+e^{(-y*h(x)})$.
I was not able to find a way to ...
- 51
2
votes
2
answers
471
views
Does Adaboost ensemble use bootstrapping?
I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as
I understand the general idea behind it: Give more weight to ...
0
votes
1
answer
122
views
How to generate a confidence interval for an adaboost prediction?
I have created a simple AdaBoostRegressor (sklearn) model which is trained on a feature set $X$ (house features) to predict a variable $y$ (house prices). The model can be used to create a prediction ...
- 23
1
vote
0
answers
102
views
What is a reasonable number of splits (maximum) for a ensemble classifier?
I am trying to use an ensemble classifier (honing in on Matlab fitcensemble). I've also explored using a single decision tree as well as tree bagging (Matlab fitctree, TreeBagger)
Simple binary (A/B) ...
- 51
3
votes
1
answer
106
views
in which part is used the splitting criteria in AdaBoost?
I have been reading the original article about AdaBoost and by comparing with other reading material it has come some doubts about this model. Please feel free to correct me if in any part I am wrong. ...
- 579
0
votes
0
answers
24
views
Getting bad results when testing my prediction model on new patients
I analyze medical diagnostic labeled data: the independent variables are medical parameters (assume that one of the variables is blood pressure) and the dependent variable is 1 or 0. The data contains ...
- 67
0
votes
0
answers
93
views
What is this symbol in Adaboost algorithm (SAMME)?
According to the original paper of SAMME algorithm (Adaboost for multiclass problems), it is described as follows:
The general idea is very straightfoward, except for this symbol:
The author didn't ...
- 133
2
votes
2
answers
713
views
Improve Adaboost that using weighted logistic regression instead of decision trees
I implemented Adaboost that using weighted logistic regression instead of decision trees and I managed to get to 0.5% error, I'm trying to improve it for days with no success and I know it possible to ...
- 21
6
votes
0
answers
78
views
Weighted Conditional Expectation definition in AdaBoost
I am looking at "Additive logistic regression a statistical view of boosting" paper (https://web.stanford.edu/~hastie/Papers/AdditiveLogisticRegression/alr.pdf)
In page 346, the authors ...
- 61
1
vote
2
answers
333
views
Adaboost -- how does reweighting affect the learning process for the subsequent learner?
In Adaboost, when you reweight the samples, how does the training process for the next classifier in the boosting algorithm take in to account the weights? Is it reflected in the loss function of the ...
- 344
2
votes
1
answer
135
views
Random forest after cross validation
i have been wondering for some time now how random forests (or AdaBoost, doesn't matter) are built when using cross-validation.
Let's see we're using 5-fold cross validation to train random forests on ...
1
vote
1
answer
115
views
Weights in Adaboost
The Adaboost algorithm is shown below.
I have a couple of questions.
First, I am a little confused on how the weights for each training sample is applied when using Adaboost. From looking at the ...
- 563
1
vote
1
answer
322
views
AdaBoost assumption of weak classifier
I've read and think I got a good grasp of the math behind AdaBoost, but I wasn't able to understand why AdaBoost requires a weak base-classifier?
Specifically, I'm dealing with AdaBoost using ...
- 113
1
vote
0
answers
28
views
Creating Predictions From Base Regressor Models in a Boosted Machine
So I'm writing my own implementation of AdaBoost using chapter 10 of Elements of Statistical Learning. The pseudocode given in the book lists how to make the algorithm for a classification problem, ...
- 297
1
vote
0
answers
119
views
AdaBoost Weight Updating for Misclassified Points
http://www.inf.fu-berlin.de/inst/ag-ki/adaboost4.pdf
I was reading this paper and came across the question showed in the second image. However, in the first image provided in a book on Machine ...
- 11
0
votes
0
answers
265
views
Why does AdaBoost use decision stumps instead of 0-depth trees?
Why is it that AdaBoost uses decision stumps for the weak learners? It seems simpler to me to just use the weighted majority of the data points for the classification. Why shouldn't we do this?
- 1
0
votes
0
answers
106
views
How to update the weight when Adaboost is used for regression
When Adaboost is used for regression, in the $m$ step, we suppose $G_m(x)$ is $m$-classifier with sample $\{(x_i,y_i)\}$; the error of i-th sample is $e^i_m;$ the weight of $e^i_m$ is $w^i_m;$ the ...
- 885
0
votes
0
answers
144
views
Termination Condition for AdaBoost.R2
I can't quite wrap my head around the termination condition of AdaBoost.R2 as defined by Drucker in this paper. On page 2 of the paper he states to "repeat the following while the average loss* $\bar{...
- 1
0
votes
1
answer
119
views
What is minimized/optimized when we use AdaBoost
When I learned about CART, we learned that at each split, we try to minimize some measure (usually Gini index) of the split. That is, we determine the predictor and threshold that decreases the Gini ...
- 2,513
3
votes
1
answer
3k
views
How do you interpret prediction output in GBM() in R for classification problem?
I created a model using the gbm() function in library(gbm). Within the gbm() function, I set the distribution as "adaboost". I have a binary response [0, 1]. I used the predict.gbm function for ...
- 2,513
1
vote
1
answer
584
views
Poor performance on Regularized models
I'm trying to build a simple model to predict the price of a cab ride, using features
such as hour, source, destination, car model, distance, and
weather features such as pressure and humidity.
I've ...
- 127
3
votes
1
answer
3k
views
Is exponential loss function the only reason for AdaBoost being adaptive algorithm?
Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to ...
- 59
0
votes
0
answers
696
views
Which distribution to choose in boosting when dataset ist binary in gbm package
I want to use the gbm package for boosting. As one can see, my data is completely binary and I'm kinda stuck which distribtion I should choose.
I'm picking all the ...
- 1
1
vote
1
answer
981
views
AdaBoost learning rate calculation
I saw the following in a Random Forrest calculation. My understanding of logarithms is not intuitive, I always have to look them up. It was asked:
...
- 111
1
vote
0
answers
1k
views
XGBoost and AdaBoostClassifier feature importances
I try to compare XGBoost and AdaBoostClassifier (from sklearn.ensemble) feature importances charts.
From this answer: https://stats.stackexchange.com/a/324418/239354 I get know that ...
- 121
1
vote
0
answers
52
views
How to combine from multiple probability in adaboost? [closed]
I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them?
For simplicity, there ...
- 11
1
vote
0
answers
711
views
Adaboost Training Error and It's Trend
The Adaboost M1 algorithm is as follows:
$\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$
with the labels $y_i \in Y = \{1,...,k\}$
weak learning algorithm WeakLearn
...
- 155
3
votes
2
answers
252
views
Adaboost Notation Confusion
The adaboost algorithm is as follows:
$\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$
with the labels $y_i \in Y = \{1,...,k\}$
weak learning algorithm WeakLearn
...
- 155