Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [adaboost]

A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.

0
votes
0answers
20 views

AdaBoost Classifier: second learner won't change decisions at all?

Recall with Adaboost, we start by giving the data points equal weight, train the first weak classifier. Then we give more weights to the data points that the first weak classifier makes mistakes and ...
0
votes
0answers
21 views

What is a population minimizer? AdaBoosting population minimizer in Elements of Statistical Learning

I am having great trouble understanding a few things related to the population minimizer on the description of AdaBoosting in the book Elements of Statistical Learning. Questions What is a ...
1
vote
0answers
12 views

How to combine from multiple probability in adaboost? [closed]

I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them? For simplicity, there ...
1
vote
0answers
23 views

Adaboost Training Error and It's Trend

The Adaboost M1 algorithm is as follows: $\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$ with the labels $y_i \in Y = \{1,...,k\}$ weak learning algorithm WeakLearn ...
3
votes
2answers
54 views

Adaboost Notation Confusion

The adaboost algorithm is as follows: $\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$ with the labels $y_i \in Y = \{1,...,k\}$ weak learning algorithm WeakLearn ...
1
vote
2answers
107 views

Binary classifiers with accuracy < 50% in Adaboost?

For a balanced binary training dataset i.e number of data points with class +1 are equal to number of data points with class -1 , what will happen if we use weak binary classifiers whose ...
0
votes
1answer
70 views

Can AdaBoost be used for regression?

I know that AdaBoost can be used for classification, but how about regression? With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) ...
1
vote
0answers
28 views

Adaboost - Show that adjusting weights brings error of current iteration to 0.5 [closed]

I'm trying to solve the following problem but I've gotten sort of stuck. So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$ and $\alpha_t = \frac{...
0
votes
0answers
25 views

ada model- variables overall importance

I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees. I would like to have a ...
0
votes
1answer
55 views

Why AdaBoost works exactly the way it does

I understand the basic idea of AdaBoost -- when training weak classifiers, use more of the difficult examples. However, it puzzles me why I sould modify the weights the way AdaBoost does. There are, ...
0
votes
0answers
31 views

Conditions for Adaboost to perform well

Under which conditions does the AdaBoost algorithm yield good results even on weak learners (i.e. slightly better than random classifiers)?
1
vote
0answers
59 views

In which cases would AdaBoost outperform Random Forest?

I have heard people claim (for example in the course Intro to machine learning , lesson 5) that they like the adaboost algorithm without really providing the reason for why. At the same time, i have ...
0
votes
0answers
15 views

Adaboost - Using Perceptrons

This is for a Image orientation project. I'm trying to implement adaboost with four perceptrons. The training data has four labels 0, 90, 180, 270. Each of the perceptron identifies one of the labels ...
0
votes
0answers
16 views

adaboost traning error based on number of training samples

There is a bound on AdaBoost training error based on the number of iterations and ϵt (weighted error of ht). Is there any boundary based on the number of training samples?
4
votes
1answer
127 views

Did I understand AdaBoost correctly?

My mantra has always been that if you are not able to recreate something you haven't really understood it. In this manner I tried to implement the AdaBoost algorithm of Freund and Schapire I used one ...
3
votes
0answers
35 views

Is Gradiant Boosting a generalization of Adaboost?

I read somewhere that Gradiant boosting is a generalization of Adaboost. However, I cannot see why. Can Anyone elaborate?
2
votes
0answers
417 views

What is the learning rate in AdaBoost? [duplicate]

In scikit-learn implementation of AdaBoost you can choose a learning rate. The documentation about AdaBoost says: "Learning rate shrinks the contribution of each classifier by learning_rate". This ...
1
vote
0answers
27 views

What stops an additional layer from destructively adding to the previous layers in an Adaboost?

Algorithmic representation of Discrete Adaboost is: Start with weights $w_i = 1/N, i = 1, .. N$ Repeat for $m = 1,...,M$ Fit the classifier $f_m(x) \in \{-1,1\}$ using weights $w_i$ on the training ...
0
votes
0answers
33 views

How to put (conditional) prediction probabilities within a range (Adaboost and general)

I am trying to understand how you can put prediction probabilities of a machine learning model within a range. Can you simply rescale it? Also why would you do this? For example, to convert the ...
0
votes
0answers
97 views

What upper boundaries make sense for the learning rate in sklearn's AdaBoostRegressor?

I'm currently trying to understand the learning rate parameter from sklearn's AdaBoostRegressor. The package only demands that the "learning_rate must be greater than zero". However, do learning rates ...
0
votes
1answer
2k views

What are the differences between XGBoost Vs AdaBoost algorithms [closed]

What are the differences between XGBoost Vs AdaBoost algorithms. Like which one can be applied under which instance.
0
votes
0answers
1k views

learning rate in Adaboost sklearn

I can't figure out what does learning_rate stand for in sklearn implementation of Adaboost. When i see the original algorithm i don't see any "learning_rate"... Meanwhile i can see from https://fr....
2
votes
1answer
75 views

How much higher accuracy of train than test is enough to consider the model overfitted?

Considering a dataset of 920 samples with 40 features in a binary classification problem. The dataset is the heart disease dataset publicly available here. I preprocessed the dataset discarding ...
4
votes
2answers
770 views

How to ensure that increasing the weights of misclassified points in AdaBoost does not adversely affect the learning progress?

It seems that we increase the weights of misclassified points on every iteration of AdaBoost. Therefore, the subsequent classifiers focus on the misclassified samples more. This would imply that these ...
1
vote
0answers
472 views

Adaboost vs Gradient Boosting Difference

Both Adaboost algorithm and Gradient Boosting (with exponential loss function) try to minimize the exponential loss function. From Elements of Statistical Learning p.344: "Hence we conclude that ...
0
votes
1answer
277 views

AdaBoost algorithm question

In the boosting algorithm,AdaBoost ,those observations which were misclassified by the classifier in the (m-1)th step have their weights increased in the mth step, and those which were correctly ...
0
votes
1answer
148 views

AdaBoostM1 reweighting examples

It is said Adaboost increases the weights of the misclassified examples. But if I look at step 2(b) , err is between 0 and 1. Then at step 2(c) , If err=1 , alpha = log(0)=-inf and if err=0, alpha = ...
2
votes
1answer
340 views

Adaboost Probabilities

Adaboost prediction is the sign of the strong classifier. How can we obtain the probability of the prediction $P(y = 1 | x)$? Can we use the logistic function or some other function as follows: $$P(...
11
votes
2answers
4k views

Boosting A Logistic Regression Model

Adaboost is an ensemble method that combines many weak learners to form a strong one. All of the examples of adaboost that i have read use decision stumps/trees as weak learners. Can i use different ...
1
vote
1answer
2k views

Feature Value Importance - AdaBoost Classifier [closed]

I'm trying to understand the impact strength of the features value's in my model. I can understand the overall feature importance based on AdaBoost's _feature_importances_ attribute. However is there ...
1
vote
0answers
15 views

Linear approach on Adaboost [closed]

For a binary classification problem, assume we have $c_1,...,c_B$ successive base classifiers obtained via AdaBoost and let $a_1,...,a_B$ their corresponding weights. Let's note the ensemble ...
1
vote
0answers
133 views

AdaBoost, gbm-package, time series [closed]

I am new at working with the gbm package. This is my R code: (data: 329 of 79 variables) ...
1
vote
0answers
311 views

Prevent AdaBoost from overfitting

I am doing this challenge on Kaggle and I am trying to use AdaBoost from scikit-learn. My code is like this: ...
1
vote
1answer
537 views

On which datasets does AdaBoost algorithm overfit?

I know that AdaBoost algorithm is less prone to overfitting but I'm curious on which kind of datasets will AdaBoost produce overfitting and why?
1
vote
0answers
73 views

Why do we increase weights of misclassified points in boosting?

I was reading this pdf and at slide at 23 I got stuck. Also how does boosting reduces the bias ? I understand it will reduce variance by averaging.
1
vote
0answers
89 views

Statistical understanding specific Adaboost algorithm modification

I'm working with the paper "Face Detection with the Modified Census Transform" Some things in this paper are not clear for me. I have written below my understanding and interpretation of the ...
1
vote
0answers
201 views

Boosting Ensemble and Support Vector Machines

Using Adaboost with SVM for classification I read the answers to the above question and got an idea of what is being talked about in the papers cited. My question is about the theoretical difference ...
1
vote
0answers
130 views

How XGBoost and Adaboost select the most important features?

I know perfectly that random forest computes the most important variables using the mean decrease Gini, but what about Adaboost and XGBoost?
0
votes
1answer
785 views

How to use ada boosting as an ensemble method in R? [closed]

I am trying to learn ensemble methods and came across that ada-boosting can be built on top of the ordinary machine learning methods such as Random forest. the method can use the misclassified data in ...
1
vote
0answers
323 views

Correct way of making a ROC curve out of n times k-fold cross-validation predictions

I wish to plot ROC curves ("ROCR" R package) of cross-validation probability predictions to compare different models obtained with Adaboost boosted tree algorithms ("gbm" R package). For instance, I ...
4
votes
1answer
6k views

Tuning adaboost

The boosting algorithm Adaboost (when using a tree) has three core parameters: number of weak learners to train learning rate max nb of splits (depth of tree) What are good practices, perhaps proven ...
3
votes
1answer
1k views

Logistic Regression + Adaboost?

In Adaboost, each sample is given a weight and the machine learning model will be trained with these weights. I want to use logistic regression model in Adaboost, but how can i use these weights in ...
2
votes
1answer
657 views

Derivation of AdaBoost.R2 algorithm

I am having difficulty understanding the derivation of the AdaBoost.R2 algorithm (AdaBoost for regression problems), as presented in this paper by Drucker (page 2), which seems to be the source that ...
1
vote
0answers
652 views

How to apply weights in building decision tree?

Hi I'm currently trying to implement Adaboost algorithm. I have implemented the weak classifier using decision tree (instead of using the fit function provided by sklearn). However, I had a tough time ...
0
votes
1answer
345 views

How do I calculate test error for adaboost?

I've calculated an adaboost algorithm for 20 iterations with a decision tree as my weak learner. I want to make a graph that plots the training error and the testing error. I have the training error,...
0
votes
1answer
51 views

Weighting the examples in AdaBoost: Distributions, gradients, and the equation

When creating a new, $j$th learner using AdaBoost, the model for defining the weight of an example is: $$w_{j}^i = e^{-y_ih_{j-1}^i}$$ These weights are created in order that the new learner will ...
2
votes
2answers
1k views

AdaBoost - Best Weak Learner with 0.5 Error

In AdaBoost, the weight of a weak learner $\alpha$ is set as $\alpha_t = \frac{1}{2}ln\frac{1-e_t}{e_t}$ under the assumptions that $e_t = \frac{1}{2} - \gamma$ and $\gamma > 0$ Therefore: $\...
4
votes
2answers
357 views

Using Bayes for combining forecasts with different accuracies (Interview question)

I have 3 independent sources for tomorrow's weather forecast: 100% probability for snow, this source is 80% accurate 50% probability for snow, this source is 60% accurate 0% probability ...
6
votes
2answers
4k views

Why is boosting less likely to overfit?

I've been learning about machine learning boosting methods (e.g., ADA boost, gradient boost) and the information sources mentioned that boosting tree methods are less likely to overfit than other ...
2
votes
1answer
1k views

Adaboost/Boosting, why the base classifier must be weak classifier?

In Boosting/Adaboost, why the base classifier must be weak classifier?