Skip to main content
18 votes

Intuitive explanations of differences between Gradient Boosting Trees (GBM) & Adaboost

An intuitive explanation of AdaBoost algorithn Let me build upon @Randel's excellent answer with an illustration of the following point In AdaBoost, ‘shortcomings’ are identified by high-weight data ...
Xavier Bourret Sicotte's user avatar
16 votes

Shrinkage parameter in Adaboost?

A visual explanation of the trade-off between learning rate and iterations This post is based on the assumption that the AdaBoost algorithm is similar to the M1 or SAMME implementations which can be ...
Xavier Bourret Sicotte's user avatar
13 votes

What is meant by 'weak learner'?

Weak learner is a learner that no matter what the distribution over the training data is will always do better than chance, when it tries to label the data. Doing better than chance means we are ...
Anish Singh Walia's user avatar
10 votes
Accepted

Boosting A Logistic Regression Model

Don't confuse the handling of the predictors (via base learners, e.g. stumps) and the handling of the loss function in boosting. Although AdaBoost can be thought of as finding combinations of base ...
EdM's user avatar
  • 94.5k
8 votes

Boosting A Logistic Regression Model

In fact we have a very similar question here on regression case. And we had a very good answer by @Matthew Drury Gradient Boosting for Linear Regression - why does it not work? Linear model (such as ...
Haitao Du's user avatar
  • 37.1k
8 votes
Accepted

Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

The main difference between AdaBoost and other "generic" boosting algorithms is that AdaBoost uses the (deviance) residuals as weights while "generic" gradient boosting algorithms use the residuals as ...
usεr11852's user avatar
  • 44.7k
7 votes
Accepted

Can decision stumps have more than 2 leaves?

You used the tag cart; CART trees always use binary splits. However, the Quinlan family of decision tree algorithms support multiple-arity splits for categorical features, and such a stump could be ...
Ben Reiniger's user avatar
  • 4,723
6 votes
Accepted

Why is boosting less likely to overfit?

The general idea is that each individual tree will over fit some parts of the data, but therefor will under fit other parts of the data. But in boosting, you don't use the individual trees, but ...
Greg Snow's user avatar
  • 52.3k
6 votes

Can AdaBoost be used for regression?

AdaBoost is a meta-algorithm, which means it can be used together with other algorithms for perfomance improvement. Indeed, the concept of boosting is a type of linear regression. Now, specifically ...
Lucas Farias's user avatar
  • 1,402
6 votes

Can decision stumps have more than 2 leaves?

@Ben Reiniger has a more complete answer with lots of useful details about random forests; definitely read that first. I, on the other hand, want to elaborate on one single point: each decision in a ...
dipetkov's user avatar
  • 10.4k
5 votes
Accepted

Feature Value Importance - AdaBoost Classifier

There are multiple ways to determine relative feature importance but as far as I know your approach might already yield the best possible results in terms of insight! AdaBoost's feature importance is ...
Lejafar's user avatar
  • 618
5 votes
Accepted

AdaBoost - Best Weak Learner with 0.5 Error

To develop the point made by Andreas: No weak learner can achieve an error rate better (i.e. lower) than 0.5 in the first round, hence it should be αt=0 for all t, making AdaBoost (with ...
Xavier Bourret Sicotte's user avatar
5 votes

Tuning adaboost

Number of weak learners Train many, many weak learners. Then look at a test-error vs. number of estimators curve to find the optimal number. Learning rate Smaller is better, but you will have to ...
Matthew Drury's user avatar
5 votes
Accepted

Adaboost/Boosting, why the base classifier must be weak classifier?

It is not required to have a "weak classifier" for base classifier, in theory you can even choose neural network as base classifier. However, weak classifier is recommended to avoid overfitting and ...
Haitao Du's user avatar
  • 37.1k
5 votes

learning rate in Adaboost sklearn

The official documentation states that "The learning rate shrinks the contribution of each regressor by learning_rate.". Thus, basically we need to understand three ...
Miguel Trejo's user avatar
5 votes

Improve Adaboost that using weighted logistic regression instead of decision trees

This seems to be pretty standard. With logistic regression as the base estimator, the adaptive boosting stops adding value after very few iterations. I put together a little notebook to illustrate, ...
Ben Reiniger's user avatar
  • 4,723
4 votes

Understand Adaboost feature selection

"Feature selection", as you describe it, boils down to fitting a depth 1 decision tree on a set of data. Given such a set and weights associated with each, the problem of fitting a decision tree (a ...
the higgs broson's user avatar
4 votes
Accepted

What does M in AdaBoost.M1 and AdaBoost.M2 stand for?

The "M" stands for "Multi-class" (Hi Ébe! I'm a little late to the party.), in contrast to binary, i.e., two-class. For example, Freund and Schapire have "First Multi-class Extension" as the title ...
K. Frank's user avatar
4 votes
Accepted

Logistic Regression + Adaboost?

Fitting to a weighted sample works the same way for pretty much any statistical/machine learning model: you minimize the sample weighted loss function. In the case of logistic regression, you would ...
Matthew Drury's user avatar
4 votes
Accepted

Derivation of AdaBoost.R2 algorithm

The choice to use the weighted median appears to be arbitrary. According to this "the idea of using the weighted median as the final regressor is not new. Freund [6] briefly mentions it and proves a ...
Pavel Komarov's user avatar
4 votes
Accepted

Adaboost Probabilities

You can do something similar, mathematically, but with a slightly different sigmoidal function to what you specified. To convert the output of AdaBoost $$F(x) = \sum_{t=1}^T \alpha_t h_t (x)$$ to a ...
A. G.'s user avatar
  • 2,151
4 votes

How to ensure that increasing the weights of misclassified points in AdaBoost does not adversely affect the learning progress?

The new classifier in each round might indeed classify the old points incorrectly. However the previous 'versions' of the classifier(from previous iterations) are not thrown away. The end result is an ...
Andreas G.'s user avatar
  • 1,465
4 votes
Accepted

AdaBoost - why decision stumps instead of trees?

The reason for using 'stumps' in boosting but full-height trees in random forests is to do with how the aggregation and fitting is done. In random forests, the trees in the ensemble are fitted ...
Thomas Lumley's user avatar
4 votes
Accepted

Boosting reduces bias when compared to what algorithm?

It is said that bagging reduces variance and boosting reduces bias. Indeed, as opposed to the base learners both ensembling methods employ. For bagging and random forests, deep/large trees are ...
Marjolein Fokkema's user avatar
3 votes
Accepted

Using Bayes for combining forecasts with different accuracies (Interview question)

With probabilistic forecasts, accuracy cannot be simply defined as (number correct forecasts)/(total number of forecasts), or better, you must define what you mean by a "correct forecast". For example ...
DeltaIV's user avatar
  • 18.1k
3 votes
Accepted

Why AdaBoost works exactly the way it does

I will justify the whole algorithm from the statistical point of view, i.e. risk minimization. I will get to the weights later. Now, just for the record, I will outline steps of the AdaBoost algorithm....
treskov's user avatar
  • 542
3 votes
Accepted

Adaboost Notation Confusion

As Lucas already said (+1), yes, your intuition is correct; that is the sum of the weighted error terms. This steps is indeed awkward for Adaboost, because it is ...
usεr11852's user avatar
  • 44.7k
3 votes
Accepted

How do you interpret prediction output in GBM() in R for classification problem?

According to gbm's reference manual: While indeed type="response" then gbm converts back to ...
usεr11852's user avatar
  • 44.7k
3 votes
Accepted

What's the purpose of learning rate in sklearn AdaBoost implementation

The learning rate is just applied to each of the tree's predictions and has nothing to do with the tree model itself but the boosting 'meta' algorithm. Since boosting is iteratively learning from the ...
Tylerr's user avatar
  • 1,562
2 votes

How to tune the weak learner in boosted algorithms

Yes, weak learners are absolutely required for boosting to be really successful. That is because each boosting round for trees actually results in more splits and a more complicated model. This will ...
Tylerr's user avatar
  • 1,562

Only top scored, non community-wiki answers of a minimum length are eligible