Questions tagged [adaboost]

A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.

Filter by
Sorted by
Tagged with
1 vote
1 answer
22 views

Boosting usa bootstraping?

I had a question about boosting. When in the first iteration of the algorithm we pass our data to the first decision tree, this data we pass is a sample generated by bootstraping or is it the original ...
user avatar
4 votes
2 answers
231 views

Can decision stumps have more than 2 leaves?

I understand decision stump: a shallow 1-level decision tree is often used as base-leaner in ensemble methods such as AdaBoost. What is not immediately clear to me ...
user avatar
  • 231
1 vote
1 answer
25 views

Divergence of accuracies and overfitting in AdaBoost

I implemented a binary classification setup of AdaBoost, but I train one model for each label in a one-vs-all arrangement, and in the prediction time I choose the class corresponding to the model with ...
user avatar
0 votes
0 answers
19 views

How AdaBoost will treat weak learners with certain errors?

I have to answer the following question and Im struggling: For each of the following cases, explain how AdaBoost will treat the weak learner $G_k$ with the weighted error $\text{err}_k(G_k, w_k) = \...
user avatar
  • 65
1 vote
0 answers
13 views

Oscillation of AdaBoost Training error

Adaboost, using weak learners as Gaussian Naive bayes, has oscillating/unpredictable training error as we increase the number of weak learners. Is there a specific reason for this? Y-axis is the ...
user avatar
  • 11
2 votes
1 answer
93 views

Boosting reduces bias when compared to what algorithm?

I am reading on bagging and boosting, and I understand how they both work (at least I think I do). I would like to talk in the context of decision tree ensembles as I think (not sure if correct) that ...
user avatar
0 votes
1 answer
20 views

How to adjust sample weights in Adasboost

I am following this video tutorial to understand Adaboost I am confused about the sample weights updating. It first calculates the amount of say of each stump by this formula, where total error is ...
user avatar
0 votes
0 answers
83 views

Confusion with AdaBoost.M1 algorithm

I'm having my head buried in the AdaBoost.M1. Are there yet some variations of AdaBoost.M1? I ask this because I read variations ...
user avatar
  • 231
0 votes
0 answers
21 views

What is the use of the learning_rate parameter in sklearn AdaBoostClassifier/Regressor?

There is a similar question here. But I have doubts regarding the answer. Hence am asking a new question clarifying my doubt. The general pattern for boosting is that the predictions of the weak ...
user avatar
3 votes
2 answers
112 views

Where is Boosting applied in Gradient boosting techniques?

In boosting, the primary idea is to re-adjust weights of training instances, so that subsequent models learn how to fit difficult-to-classify samples. From Wikipedia Boosting (Machine Learning): ...
user avatar
1 vote
1 answer
28 views

how train AdaBoost M1 weak estimators?

I'm trying to implement AdaBoost.M1 as explained in Boosting: Foundations and Algorithms by Robert E. Schapire and Y. Freund. The problem is that I don't understand at each iteration t the estimator ...
user avatar
0 votes
0 answers
45 views

Equation 10.2 from The Elements of Statistical Learning. Median of a chi-squared distribution

I'm reading about AdaBoost in the The Elements of Statistical Learning and I don't understand the equation 10.2. Below is an excerpt from the book. The power of AdaBoost to dramatically increase the ...
user avatar
0 votes
1 answer
60 views

Prediction of Adaboost Classifier between 0 and 1

I have to use an Adaboost classifier to predict if the data is a signal (1) or a background (0) event. But since the output is expected to be 1 or 0 the ...
user avatar
2 votes
0 answers
65 views

Generalisation error bound proof for the Adaboost algorithm, corollary 4.4 in Freund and Schapire (2014)

I am having difficulty with a detail in the proof of generalisation error bounds for the AdaBoost algorithm, and would appreciate some assistance. This is the proof of corollary 4.4 on page 54 of ...
user avatar
  • 2,206
3 votes
0 answers
165 views

Why does classifier (XGBoost) "after PCA" runtime increase compared to "before PCA"

The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
user avatar
0 votes
0 answers
132 views

How does resampling in AdaBoost (exactly) work?

Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
user avatar
  • 125
1 vote
1 answer
667 views

What's the purpose of learning rate in sklearn AdaBoost implementation

We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner. Conceptually, ...
user avatar
  • 231
1 vote
0 answers
57 views

Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?

I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
user avatar
1 vote
0 answers
45 views

Interpretation of a tightly paired learning curve with increasing loss

I am assessing models for a binary classification task and have created a model with a very strange learning curve. This is the learning curve of an sklearn AdaBoostClassifier fitted with default ...
user avatar
  • 11
0 votes
1 answer
69 views

Why is AdaBoost with short decision trees a form of feature selection?

Why is AdaBoost with short decision trees a form of feature selection? What is so special with short decision trees?
user avatar
2 votes
2 answers
1k views

AdaBoost - why decision stumps instead of trees?

Since the original AdaBoost article it has been found out that boosting reduces both variance and bias in the classifier (in contrast to bagging, which only reduces variance). Original AdaBoost (and ...
user avatar
  • 179
1 vote
1 answer
79 views

Why does AdaBoost algorithm use weighted data points?

I am learning about AdaBoost algorithm. At each iteration, adaBoost set higher weight to mistake datapoints, and lower weight to correct classified data points. I do not understand why the algorithm ...
user avatar
1 vote
0 answers
23 views

what will be modified algorithm if this loss function will be modified? [closed]

I have a loss function of AdaBoost and $l(h(x),y)=e^{(-y*h(x))}$ and what will be modified algorithm if we modify loss function to $l(h(x),y)=ln(1+e^{(-y*h(x)})$. I was not able to find a way to ...
user avatar
2 votes
2 answers
471 views

Does Adaboost ensemble use bootstrapping?

I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as I understand the general idea behind it: Give more weight to ...
user avatar
0 votes
1 answer
122 views

How to generate a confidence interval for an adaboost prediction?

I have created a simple AdaBoostRegressor (sklearn) model which is trained on a feature set $X$ (house features) to predict a variable $y$ (house prices). The model can be used to create a prediction ...
user avatar
  • 23
1 vote
0 answers
102 views

What is a reasonable number of splits (maximum) for a ensemble classifier?

I am trying to use an ensemble classifier (honing in on Matlab fitcensemble). I've also explored using a single decision tree as well as tree bagging (Matlab fitctree, TreeBagger) Simple binary (A/B) ...
user avatar
3 votes
1 answer
106 views

in which part is used the splitting criteria in AdaBoost?

I have been reading the original article about AdaBoost and by comparing with other reading material it has come some doubts about this model. Please feel free to correct me if in any part I am wrong. ...
user avatar
  • 579
0 votes
0 answers
24 views

Getting bad results when testing my prediction model on new patients

I analyze medical diagnostic labeled data: the independent variables are medical parameters (assume that one of the variables is blood pressure) and the dependent variable is 1 or 0. The data contains ...
user avatar
  • 67
0 votes
0 answers
93 views

What is this symbol in Adaboost algorithm (SAMME)?

According to the original paper of SAMME algorithm (Adaboost for multiclass problems), it is described as follows: The general idea is very straightfoward, except for this symbol: The author didn't ...
user avatar
  • 133
2 votes
2 answers
713 views

Improve Adaboost that using weighted logistic regression instead of decision trees

I implemented Adaboost that using weighted logistic regression instead of decision trees and I managed to get to 0.5% error, I'm trying to improve it for days with no success and I know it possible to ...
user avatar
6 votes
0 answers
78 views

Weighted Conditional Expectation definition in AdaBoost

I am looking at "Additive logistic regression a statistical view of boosting" paper (https://web.stanford.edu/~hastie/Papers/AdditiveLogisticRegression/alr.pdf) In page 346, the authors ...
user avatar
1 vote
2 answers
333 views

Adaboost -- how does reweighting affect the learning process for the subsequent learner?

In Adaboost, when you reweight the samples, how does the training process for the next classifier in the boosting algorithm take in to account the weights? Is it reflected in the loss function of the ...
user avatar
2 votes
1 answer
135 views

Random forest after cross validation

i have been wondering for some time now how random forests (or AdaBoost, doesn't matter) are built when using cross-validation. Let's see we're using 5-fold cross validation to train random forests on ...
user avatar
1 vote
1 answer
115 views

Weights in Adaboost

The Adaboost algorithm is shown below. I have a couple of questions. First, I am a little confused on how the weights for each training sample is applied when using Adaboost. From looking at the ...
user avatar
  • 563
1 vote
1 answer
322 views

AdaBoost assumption of weak classifier

I've read and think I got a good grasp of the math behind AdaBoost, but I wasn't able to understand why AdaBoost requires a weak base-classifier? Specifically, I'm dealing with AdaBoost using ...
user avatar
  • 113
1 vote
0 answers
28 views

Creating Predictions From Base Regressor Models in a Boosted Machine

So I'm writing my own implementation of AdaBoost using chapter 10 of Elements of Statistical Learning. The pseudocode given in the book lists how to make the algorithm for a classification problem, ...
user avatar
1 vote
0 answers
119 views

AdaBoost Weight Updating for Misclassified Points

http://www.inf.fu-berlin.de/inst/ag-ki/adaboost4.pdf I was reading this paper and came across the question showed in the second image. However, in the first image provided in a book on Machine ...
user avatar
0 votes
0 answers
265 views

Why does AdaBoost use decision stumps instead of 0-depth trees?

Why is it that AdaBoost uses decision stumps for the weak learners? It seems simpler to me to just use the weighted majority of the data points for the classification. Why shouldn't we do this?
user avatar
0 votes
0 answers
106 views

How to update the weight when Adaboost is used for regression

When Adaboost is used for regression, in the $m$ step, we suppose $G_m(x)$ is $m$-classifier with sample $\{(x_i,y_i)\}$; the error of i-th sample is $e^i_m;$ the weight of $e^i_m$ is $w^i_m;$ the ...
user avatar
0 votes
0 answers
144 views

Termination Condition for AdaBoost.R2

I can't quite wrap my head around the termination condition of AdaBoost.R2 as defined by Drucker in this paper. On page 2 of the paper he states to "repeat the following while the average loss* $\bar{...
user avatar
0 votes
1 answer
119 views

What is minimized/optimized when we use AdaBoost

When I learned about CART, we learned that at each split, we try to minimize some measure (usually Gini index) of the split. That is, we determine the predictor and threshold that decreases the Gini ...
user avatar
  • 2,513
3 votes
1 answer
3k views

How do you interpret prediction output in GBM() in R for classification problem?

I created a model using the gbm() function in library(gbm). Within the gbm() function, I set the distribution as "adaboost". I have a binary response [0, 1]. I used the predict.gbm function for ...
user avatar
  • 2,513
1 vote
1 answer
584 views

Poor performance on Regularized models

I'm trying to build a simple model to predict the price of a cab ride, using features such as hour, source, destination, car model, distance, and weather features such as pressure and humidity. I've ...
user avatar
  • 127
3 votes
1 answer
3k views

Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to ...
user avatar
0 votes
0 answers
696 views

Which distribution to choose in boosting when dataset ist binary in gbm package

I want to use the gbm package for boosting. As one can see, my data is completely binary and I'm kinda stuck which distribtion I should choose. I'm picking all the ...
user avatar
  • 1
1 vote
1 answer
981 views

AdaBoost learning rate calculation

I saw the following in a Random Forrest calculation. My understanding of logarithms is not intuitive, I always have to look them up. It was asked: ...
user avatar
1 vote
0 answers
1k views

XGBoost and AdaBoostClassifier feature importances

I try to compare XGBoost and AdaBoostClassifier (from sklearn.ensemble) feature importances charts. From this answer: https://stats.stackexchange.com/a/324418/239354 I get know that ...
user avatar
  • 121
1 vote
0 answers
52 views

How to combine from multiple probability in adaboost? [closed]

I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them? For simplicity, there ...
user avatar
  • 11
1 vote
0 answers
711 views

Adaboost Training Error and It's Trend

The Adaboost M1 algorithm is as follows: $\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$ with the labels $y_i \in Y = \{1,...,k\}$ weak learning algorithm WeakLearn ...
user avatar
  • 155
3 votes
2 answers
252 views

Adaboost Notation Confusion

The adaboost algorithm is as follows: $\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$ with the labels $y_i \in Y = \{1,...,k\}$ weak learning algorithm WeakLearn ...
user avatar
  • 155