Questions tagged [adaboost]
A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.
85
questions
0
votes
0answers
54 views
Learning rate constant in Random Forrest calculation
I saw the following in a Random Forrest calculation. My understanding of logarithms is not intuitive, I always have to look them up. It was asked:
...
1
vote
0answers
76 views
XGBoost and AdaBoostClassifier feature importances
I try to compare XGBoost and AdaBoostClassifier (from sklearn.ensemble) feature importances charts.
From this answer: https://stats.stackexchange.com/a/324418/239354 I get know that ...
0
votes
0answers
24 views
AdaBoost Classifier: second learner won't change decisions at all?
Recall with Adaboost, we start by giving the data points equal weight, train the first weak classifier. Then we give more weights to the data points that the first weak classifier makes mistakes and ...
0
votes
0answers
26 views
What is a population minimizer? AdaBoosting population minimizer in Elements of Statistical Learning
I am having great trouble understanding a few things related to the population minimizer on the description of AdaBoosting in the book Elements of Statistical Learning.
Questions
What is a ...
1
vote
0answers
16 views
How to combine from multiple probability in adaboost? [closed]
I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them?
For simplicity, there ...
1
vote
0answers
34 views
Adaboost Training Error and It's Trend
The Adaboost M1 algorithm is as follows:
$\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$
with the labels $y_i \in Y = \{1,...,k\}$
weak learning algorithm WeakLearn
...
3
votes
2answers
60 views
Adaboost Notation Confusion
The adaboost algorithm is as follows:
$\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$
with the labels $y_i \in Y = \{1,...,k\}$
weak learning algorithm WeakLearn
...
1
vote
2answers
131 views
Binary classifiers with accuracy < 50% in Adaboost?
For a balanced binary training dataset i.e number of data points with class +1 are equal to number of data points with class -1 , what will happen if we use weak binary classifiers whose ...
0
votes
1answer
241 views
Can AdaBoost be used for regression?
I know that AdaBoost can be used for classification, but how about regression?
With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) ...
1
vote
0answers
30 views
Adaboost - Show that adjusting weights brings error of current iteration to 0.5 [closed]
I'm trying to solve the following problem but I've gotten sort of stuck.
So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$
and $\alpha_t = \frac{...
0
votes
0answers
26 views
ada model- variables overall importance
I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees.
I would like to have a ...
0
votes
1answer
76 views
Why AdaBoost works exactly the way it does
I understand the basic idea of AdaBoost -- when training weak classifiers, use more of the difficult examples. However, it puzzles me why I sould modify the weights the way AdaBoost does. There are, ...
0
votes
0answers
47 views
Conditions for Adaboost to perform well
Under which conditions does the AdaBoost algorithm yield good results even on weak learners (i.e. slightly better than random classifiers)?
1
vote
0answers
110 views
In which cases would AdaBoost outperform Random Forest?
I have heard people claim (for example in the course Intro to machine learning , lesson 5) that they like the adaboost algorithm without really providing the reason for why.
At the same time, i have ...
0
votes
0answers
18 views
Adaboost - Using Perceptrons
This is for a Image orientation project. I'm trying to implement adaboost with four perceptrons. The training data has four labels 0, 90, 180, 270. Each of the perceptron identifies one of the labels ...
0
votes
0answers
17 views
adaboost traning error based on number of training samples
There is a bound on AdaBoost training error based on the number of iterations and ϵt (weighted error of ht). Is there any boundary based on the number of training samples?
4
votes
1answer
143 views
Did I understand AdaBoost correctly?
My mantra has always been that if you are not able to recreate something you haven't really understood it. In this manner I tried to implement the AdaBoost algorithm of Freund and Schapire
I used one ...
3
votes
0answers
37 views
Is Gradiant Boosting a generalization of Adaboost?
I read somewhere that Gradiant boosting is a generalization of Adaboost. However, I cannot see why. Can Anyone elaborate?
2
votes
0answers
607 views
What is the learning rate in AdaBoost? [duplicate]
In scikit-learn implementation of AdaBoost you can choose a learning rate.
The documentation about AdaBoost says:
"Learning rate shrinks the contribution of each classifier by learning_rate".
This ...
1
vote
0answers
27 views
What stops an additional layer from destructively adding to the previous layers in an Adaboost?
Algorithmic representation of Discrete Adaboost is:
Start with weights $w_i = 1/N, i = 1, .. N$
Repeat for $m = 1,...,M$
Fit the classifier $f_m(x) \in \{-1,1\}$ using weights $w_i$ on the training ...
0
votes
0answers
1k views
learning rate in Adaboost sklearn
I can't figure out what does learning_rate stand for in sklearn implementation of Adaboost. When i see the original algorithm i don't see any "learning_rate"...
Meanwhile i can see from https://fr....
2
votes
1answer
79 views
How much higher accuracy of train than test is enough to consider the model overfitted?
Considering a dataset of 920 samples with 40 features in a binary classification problem. The dataset is the heart disease dataset publicly available here.
I preprocessed the dataset discarding ...
4
votes
2answers
844 views
How to ensure that increasing the weights of misclassified points in AdaBoost does not adversely affect the learning progress?
It seems that we increase the weights of misclassified points on every iteration of AdaBoost. Therefore, the subsequent classifiers focus on the misclassified samples more. This would imply that these ...
1
vote
0answers
498 views
Adaboost vs Gradient Boosting Difference
Both Adaboost algorithm and Gradient Boosting (with exponential loss function) try to minimize the exponential loss function.
From Elements of Statistical Learning p.344: "Hence
we conclude that ...
0
votes
1answer
298 views
AdaBoost algorithm question
In the boosting algorithm,AdaBoost ,those observations which were misclassified by the classifier in the (m-1)th step have their weights increased in the mth step, and those which were correctly ...
0
votes
1answer
170 views
AdaBoostM1 reweighting examples
It is said Adaboost increases the weights of the misclassified examples. But if I look at step 2(b) , err is between 0 and 1. Then at step 2(c) , If err=1 , alpha = log(0)=-inf and if err=0, alpha = ...
2
votes
1answer
385 views
Adaboost Probabilities
Adaboost prediction is the sign of the strong classifier. How can we obtain the probability of the prediction $P(y = 1 | x)$?
Can we use the logistic function or some other function as follows:
$$P(...
11
votes
2answers
5k views
Boosting A Logistic Regression Model
Adaboost is an ensemble method that combines many weak learners to form a strong one. All of the examples of adaboost that i have read use decision stumps/trees as weak learners. Can i use different ...
1
vote
1answer
2k views
Feature Value Importance - AdaBoost Classifier [closed]
I'm trying to understand the impact strength of the features value's in my model.
I can understand the overall feature importance based on AdaBoost's _feature_importances_ attribute. However is there ...
1
vote
0answers
16 views
Linear approach on Adaboost [closed]
For a binary classification problem, assume we have $c_1,...,c_B$ successive base classifiers obtained via AdaBoost and let $a_1,...,a_B$ their corresponding weights. Let's note the ensemble ...
1
vote
0answers
139 views
AdaBoost, gbm-package, time series [closed]
I am new at working with the gbm package.
This is my R code: (data: 329 of 79 variables)
...
1
vote
0answers
336 views
Prevent AdaBoost from overfitting
I am doing this challenge on Kaggle and I am trying to use AdaBoost from scikit-learn.
My code is like this:
...
1
vote
1answer
603 views
On which datasets does AdaBoost algorithm overfit?
I know that AdaBoost algorithm is less prone to overfitting but I'm curious on which kind of datasets will AdaBoost produce overfitting and why?
1
vote
0answers
78 views
Why do we increase weights of misclassified points in boosting?
I was reading this pdf and at slide at 23 I got stuck.
Also how does boosting reduces the bias ? I understand it will reduce variance by averaging.
1
vote
0answers
91 views
Statistical understanding specific Adaboost algorithm modification
I'm working with the paper "Face Detection with the Modified Census Transform"
Some things in this paper are not clear for me.
I have written below my understanding and interpretation of the ...
1
vote
0answers
207 views
Boosting Ensemble and Support Vector Machines
Using Adaboost with SVM for classification
I read the answers to the above question and got an idea of what is being talked about in the papers cited. My question is about the theoretical difference ...
1
vote
0answers
141 views
How XGBoost and Adaboost select the most important features?
I know perfectly that random forest computes the most important variables using the mean decrease Gini, but what about Adaboost and XGBoost?
0
votes
1answer
826 views
How to use ada boosting as an ensemble method in R? [closed]
I am trying to learn ensemble methods and came across that ada-boosting can be built on top of the ordinary machine learning methods such as Random forest. the method can use the misclassified data in ...
1
vote
0answers
337 views
Correct way of making a ROC curve out of n times k-fold cross-validation predictions
I wish to plot ROC curves ("ROCR" R package) of cross-validation probability predictions to compare different models obtained with Adaboost boosted tree algorithms ("gbm" R package).
For instance, I ...
4
votes
1answer
6k views
Tuning adaboost
The boosting algorithm Adaboost (when using a tree) has three core parameters:
number of weak learners to train
learning rate
max nb of splits (depth of tree)
What are good practices, perhaps proven ...
3
votes
1answer
2k views
Logistic Regression + Adaboost?
In Adaboost, each sample is given a weight and the machine learning model will be trained with these weights. I want to use logistic regression model in Adaboost, but how can i use these weights in ...
2
votes
1answer
728 views
Derivation of AdaBoost.R2 algorithm
I am having difficulty understanding the derivation of the AdaBoost.R2 algorithm (AdaBoost for regression problems), as presented in this paper by Drucker (page 2), which seems to be the source that ...
1
vote
0answers
716 views
How to apply weights in building decision tree?
Hi I'm currently trying to implement Adaboost algorithm. I have implemented the weak classifier using decision tree (instead of using the fit function provided by sklearn). However, I had a tough time ...
0
votes
1answer
357 views
How do I calculate test error for adaboost?
I've calculated an adaboost algorithm for 20 iterations with a decision tree as my weak learner. I want to make a graph that plots the training error and the testing error. I have the training error,...
0
votes
1answer
51 views
Weighting the examples in AdaBoost: Distributions, gradients, and the equation
When creating a new, $j$th learner using AdaBoost, the model for defining the weight of an example is:
$$w_{j}^i = e^{-y_ih_{j-1}^i}$$
These weights are created in order that the new learner will ...
2
votes
2answers
1k views
AdaBoost - Best Weak Learner with 0.5 Error
In AdaBoost, the weight of a weak learner $\alpha$ is set as
$\alpha_t = \frac{1}{2}ln\frac{1-e_t}{e_t}$
under the assumptions that
$e_t = \frac{1}{2} - \gamma$ and $\gamma > 0$
Therefore:
$\...
4
votes
2answers
365 views
Using Bayes for combining forecasts with different accuracies (Interview question)
I have 3 independent sources for tomorrow's weather forecast:
100% probability for snow, this source is 80% accurate
50% probability for snow, this source is 60% accurate
0% probability ...
6
votes
2answers
5k views
Why is boosting less likely to overfit?
I've been learning about machine learning boosting methods (e.g., ADA boost, gradient boost) and the information sources mentioned that boosting tree methods are less likely to overfit than other ...
2
votes
1answer
1k views
Adaboost/Boosting, why the base classifier must be weak classifier?
In Boosting/Adaboost, why the base classifier must be weak classifier?
1
vote
0answers
257 views
Why would (ada) boosting increase classification error for multinomial logistic regression model?
A basic multinomial logistic regression model doesn't do great (~20%) in terms of test classification error for my problem, so a thought I had right away was to apply the adaboost to lower that. ...
