Questions tagged [adaboost]
A popular boosting algorithm (short for "adaptive boosting"). Boosting combines weakly predictive models into a strongly predictive model.
85
questions
1
vote
1answer
46 views
Poor performance on Regularized models
I'm trying to build a simple model to predict the price of a cab ride, using features
such as hour, source, destination, car model, distance, and
weather features such as pressure and humidity.
I've ...
0
votes
0answers
15 views
Generalization error of AdaBoost in the context of PAC-learning theory
Let $\mathcal{X}$ be a non-empty set and $\mathcal{Y}:=\{-1,+1\}$ and $\emptyset\neq \mathcal{H}\subset\mathcal{Y}^\mathcal{X}$.
Recall the AdaBoost algorithm.
Input: a sample $s_m:=\left((x_1,y_1),....
0
votes
0answers
16 views
Adabooster Feature Selection?
Hi i am looking for some guidance. I am creating a series of models, my first was a basic decision tree classifier, my second was a random forest classifier and my third is an Ada boost. With my other ...
3
votes
1answer
148 views
Is exponential loss function the only reason for AdaBoost being adaptive algorithm?
Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to ...
0
votes
0answers
43 views
Which distribution to choose in boosting when dataset ist binary in gbm package
I want to use the gbm package for boosting. As one can see, my data is completely binary and I'm kinda stuck which distribtion I should choose.
I'm picking all the ...
1
vote
1answer
126 views
AdaBoost learning rate calculation
I saw the following in a Random Forrest calculation. My understanding of logarithms is not intuitive, I always have to look them up. It was asked:
...
1
vote
0answers
171 views
XGBoost and AdaBoostClassifier feature importances
I try to compare XGBoost and AdaBoostClassifier (from sklearn.ensemble) feature importances charts.
From this answer: https://stats.stackexchange.com/a/324418/239354 I get know that ...
0
votes
0answers
48 views
What is a population minimizer? AdaBoosting population minimizer in Elements of Statistical Learning
I am having great trouble understanding a few things related to the population minimizer on the description of AdaBoosting in the book Elements of Statistical Learning.
Questions
What is a ...
1
vote
0answers
24 views
How to combine from multiple probability in adaboost? [closed]
I tried to implement adaboost, then I want to create ROC and count for the AUROC. I use tree as my base classifier. I got the probability from each tree. How to combine them?
For simplicity, there ...
1
vote
0answers
96 views
Adaboost Training Error and It's Trend
The Adaboost M1 algorithm is as follows:
$\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$
with the labels $y_i \in Y = \{1,...,k\}$
weak learning algorithm WeakLearn
...
3
votes
2answers
99 views
Adaboost Notation Confusion
The adaboost algorithm is as follows:
$\mathbf{Input}$: sequence of m examples $<(x_1,y_1),...,(x_m,y_m)>$
with the labels $y_i \in Y = \{1,...,k\}$
weak learning algorithm WeakLearn
...
1
vote
2answers
282 views
Binary classifiers with accuracy < 50% in Adaboost?
For a balanced binary training dataset i.e number of data points with class +1 are equal to number of data points with class -1 , what will happen if we use weak binary classifiers whose ...
2
votes
1answer
2k views
Can AdaBoost be used for regression?
I know that AdaBoost can be used for classification, but how about regression?
With classification, it is clear how to assign the "amount of say" (or weight) to the predictions of each model (stump) ...
1
vote
0answers
42 views
Adaboost - Show that adjusting weights brings error of current iteration to 0.5 [closed]
I'm trying to solve the following problem but I've gotten sort of stuck.
So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$
and $\alpha_t = \frac{...
0
votes
0answers
28 views
ada model- variables overall importance
I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees.
I would like to have a ...
0
votes
1answer
104 views
Why AdaBoost works exactly the way it does
I understand the basic idea of AdaBoost -- when training weak classifiers, use more of the difficult examples. However, it puzzles me why I sould modify the weights the way AdaBoost does. There are, ...
0
votes
0answers
54 views
Conditions for Adaboost to perform well
Under which conditions does the AdaBoost algorithm yield good results even on weak learners (i.e. slightly better than random classifiers)?
1
vote
0answers
289 views
In which cases would AdaBoost outperform Random Forest?
I have heard people claim (for example in the course Intro to machine learning , lesson 5) that they like the adaboost algorithm without really providing the reason for why.
At the same time, i have ...
4
votes
1answer
192 views
Did I understand AdaBoost correctly?
My mantra has always been that if you are not able to recreate something you haven't really understood it. In this manner I tried to implement the AdaBoost algorithm of Freund and Schapire
I used one ...
3
votes
0answers
51 views
Is Gradiant Boosting a generalization of Adaboost?
I read somewhere that Gradiant boosting is a generalization of Adaboost. However, I cannot see why. Can Anyone elaborate?
2
votes
0answers
1k views
What is the learning rate in AdaBoost? [duplicate]
In scikit-learn implementation of AdaBoost you can choose a learning rate.
The documentation about AdaBoost says:
"Learning rate shrinks the contribution of each classifier by learning_rate".
This ...
1
vote
0answers
27 views
What stops an additional layer from destructively adding to the previous layers in an Adaboost?
Algorithmic representation of Discrete Adaboost is:
Start with weights $w_i = 1/N, i = 1, .. N$
Repeat for $m = 1,...,M$
Fit the classifier $f_m(x) \in \{-1,1\}$ using weights $w_i$ on the training ...
1
vote
1answer
2k views
learning rate in Adaboost sklearn
I can't figure out what does learning_rate stand for in sklearn implementation of Adaboost. When i see the original algorithm i don't see any "learning_rate"...
Meanwhile i can see from https://fr....
2
votes
1answer
98 views
How much higher accuracy of train than test is enough to consider the model overfitted?
Considering a dataset of 920 samples with 40 features in a binary classification problem. The dataset is the heart disease dataset publicly available here.
I preprocessed the dataset discarding ...
4
votes
2answers
1k views
How to ensure that increasing the weights of misclassified points in AdaBoost does not adversely affect the learning progress?
It seems that we increase the weights of misclassified points on every iteration of AdaBoost. Therefore, the subsequent classifiers focus on the misclassified samples more. This would imply that these ...
1
vote
0answers
543 views
Adaboost vs Gradient Boosting Difference
Both Adaboost algorithm and Gradient Boosting (with exponential loss function) try to minimize the exponential loss function.
From Elements of Statistical Learning p.344: "Hence
we conclude that ...
0
votes
1answer
377 views
AdaBoost algorithm question
In the boosting algorithm,AdaBoost ,those observations which were misclassified by the classifier in the (m-1)th step have their weights increased in the mth step, and those which were correctly ...
0
votes
1answer
247 views
AdaBoostM1 reweighting examples
It is said Adaboost increases the weights of the misclassified examples. But if I look at step 2(b) , err is between 0 and 1. Then at step 2(c) , If err=1 , alpha = log(0)=-inf and if err=0, alpha = ...
2
votes
1answer
506 views
Adaboost Probabilities
Adaboost prediction is the sign of the strong classifier. How can we obtain the probability of the prediction $P(y = 1 | x)$?
Can we use the logistic function or some other function as follows:
$$P(...
11
votes
2answers
6k views
Boosting A Logistic Regression Model
Adaboost is an ensemble method that combines many weak learners to form a strong one. All of the examples of adaboost that i have read use decision stumps/trees as weak learners. Can i use different ...
1
vote
1answer
3k views
Feature Value Importance - AdaBoost Classifier [closed]
I'm trying to understand the impact strength of the features value's in my model.
I can understand the overall feature importance based on AdaBoost's _feature_importances_ attribute. However is there ...
1
vote
0answers
414 views
Prevent AdaBoost from overfitting
I am doing this challenge on Kaggle and I am trying to use AdaBoost from scikit-learn.
My code is like this:
...
0
votes
1answer
716 views
On which datasets does AdaBoost algorithm overfit? [closed]
I know that AdaBoost algorithm is less prone to overfitting but I'm curious on which kind of datasets will AdaBoost produce overfitting and why?
1
vote
0answers
97 views
Why do we increase weights of misclassified points in boosting?
I was reading this pdf and at slide at 23 I got stuck.
Also how does boosting reduces the bias ? I understand it will reduce variance by averaging.
1
vote
0answers
95 views
Statistical understanding specific Adaboost algorithm modification
I'm working with the paper "Face Detection with the Modified Census Transform"
Some things in this paper are not clear for me.
I have written below my understanding and interpretation of the ...
1
vote
0answers
226 views
Boosting Ensemble and Support Vector Machines
Using Adaboost with SVM for classification
I read the answers to the above question and got an idea of what is being talked about in the papers cited. My question is about the theoretical difference ...
1
vote
0answers
151 views
How XGBoost and Adaboost select the most important features?
I know perfectly that random forest computes the most important variables using the mean decrease Gini, but what about Adaboost and XGBoost?
0
votes
1answer
1k views
How to use ada boosting as an ensemble method in R? [closed]
I am trying to learn ensemble methods and came across that ada-boosting can be built on top of the ordinary machine learning methods such as Random forest. the method can use the misclassified data in ...
1
vote
0answers
368 views
Correct way of making a ROC curve out of n times k-fold cross-validation predictions
I wish to plot ROC curves ("ROCR" R package) of cross-validation probability predictions to compare different models obtained with Adaboost boosted tree algorithms ("gbm" R package).
For instance, I ...
4
votes
1answer
8k views
Tuning adaboost
The boosting algorithm Adaboost (when using a tree) has three core parameters:
number of weak learners to train
learning rate
max nb of splits (depth of tree)
What are good practices, perhaps proven ...
3
votes
1answer
2k views
Logistic Regression + Adaboost?
In Adaboost, each sample is given a weight and the machine learning model will be trained with these weights. I want to use logistic regression model in Adaboost, but how can i use these weights in ...
2
votes
1answer
931 views
Derivation of AdaBoost.R2 algorithm
I am having difficulty understanding the derivation of the AdaBoost.R2 algorithm (AdaBoost for regression problems), as presented in this paper by Drucker (page 2), which seems to be the source that ...
1
vote
0answers
981 views
How to apply weights in building decision tree?
Hi I'm currently trying to implement Adaboost algorithm. I have implemented the weak classifier using decision tree (instead of using the fit function provided by sklearn). However, I had a tough time ...
0
votes
1answer
393 views
How do I calculate test error for adaboost?
I've calculated an adaboost algorithm for 20 iterations with a decision tree as my weak learner. I want to make a graph that plots the training error and the testing error. I have the training error,...
0
votes
1answer
53 views
Weighting the examples in AdaBoost: Distributions, gradients, and the equation
When creating a new, $j$th learner using AdaBoost, the model for defining the weight of an example is:
$$w_{j}^i = e^{-y_ih_{j-1}^i}$$
These weights are created in order that the new learner will ...
2
votes
2answers
2k views
AdaBoost - Best Weak Learner with 0.5 Error
In AdaBoost, the weight of a weak learner $\alpha$ is set as
$\alpha_t = \frac{1}{2}ln\frac{1-e_t}{e_t}$
under the assumptions that
$e_t = \frac{1}{2} - \gamma$ and $\gamma > 0$
Therefore:
$\...
4
votes
2answers
401 views
Using Bayes for combining forecasts with different accuracies (Interview question)
I have 3 independent sources for tomorrow's weather forecast:
100% probability for snow, this source is 80% accurate
50% probability for snow, this source is 60% accurate
0% probability ...
6
votes
2answers
6k views
Why is boosting less likely to overfit?
I've been learning about machine learning boosting methods (e.g., ADA boost, gradient boost) and the information sources mentioned that boosting tree methods are less likely to overfit than other ...
2
votes
1answer
1k views
Adaboost/Boosting, why the base classifier must be weak classifier?
In Boosting/Adaboost, why the base classifier must be weak classifier?
1
vote
1answer
326 views
Why would (ada) boosting increase classification error for multinomial logistic regression model? [closed]
A basic multinomial logistic regression model doesn't do great (~20%) in terms of test classification error for my problem, so a thought I had right away was to apply the adaboost to lower that. ...
