# Tag Info

### Intuitive explanations of differences between Gradient Boosting Trees (GBM) & Adaboost

An intuitive explanation of AdaBoost algorithn Let me build upon @Randel's excellent answer with an illustration of the following point In AdaBoost, ‘shortcomings’ are identified by high-weight data ...

A visual explanation of the trade-off between learning rate and iterations This post is based on the assumption that the AdaBoost algorithm is similar to the M1 or SAMME implementations which can be ...

### What is meant by 'weak learner'?

Weak learner is a learner that no matter what the distribution over the training data is will always do better than chance, when it tries to label the data. Doing better than chance means we are ...
Accepted

### Boosting A Logistic Regression Model

Don't confuse the handling of the predictors (via base learners, e.g. stumps) and the handling of the loss function in boosting. Although AdaBoost can be thought of as finding combinations of base ...
• 94.5k

### Boosting A Logistic Regression Model

In fact we have a very similar question here on regression case. And we had a very good answer by @Matthew Drury Gradient Boosting for Linear Regression - why does it not work? Linear model (such as ...
• 37.1k
Accepted

### Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

The main difference between AdaBoost and other "generic" boosting algorithms is that AdaBoost uses the (deviance) residuals as weights while "generic" gradient boosting algorithms use the residuals as ...
• 44.7k
Accepted

### Can decision stumps have more than 2 leaves?

You used the tag cart; CART trees always use binary splits. However, the Quinlan family of decision tree algorithms support multiple-arity splits for categorical features, and such a stump could be ...
• 4,723
Accepted

### Why is boosting less likely to overfit?

The general idea is that each individual tree will over fit some parts of the data, but therefor will under fit other parts of the data. But in boosting, you don't use the individual trees, but ...
• 52.3k

### Can AdaBoost be used for regression?

AdaBoost is a meta-algorithm, which means it can be used together with other algorithms for perfomance improvement. Indeed, the concept of boosting is a type of linear regression. Now, specifically ...
• 1,402

### Can decision stumps have more than 2 leaves?

@Ben Reiniger has a more complete answer with lots of useful details about random forests; definitely read that first. I, on the other hand, want to elaborate on one single point: each decision in a ...
• 10.4k
Accepted

### Feature Value Importance - AdaBoost Classifier

There are multiple ways to determine relative feature importance but as far as I know your approach might already yield the best possible results in terms of insight! AdaBoost's feature importance is ...
• 618
Accepted

### AdaBoost - Best Weak Learner with 0.5 Error

To develop the point made by Andreas: No weak learner can achieve an error rate better (i.e. lower) than 0.5 in the first round, hence it should be αt=0 for all t, making AdaBoost (with ...

Number of weak learners Train many, many weak learners. Then look at a test-error vs. number of estimators curve to find the optimal number. Learning rate Smaller is better, but you will have to ...
Accepted

### Adaboost/Boosting, why the base classifier must be weak classifier?

It is not required to have a "weak classifier" for base classifier, in theory you can even choose neural network as base classifier. However, weak classifier is recommended to avoid overfitting and ...
• 37.1k

### learning rate in Adaboost sklearn

The official documentation states that "The learning rate shrinks the contribution of each regressor by learning_rate.". Thus, basically we need to understand three ...
• 301

### Improve Adaboost that using weighted logistic regression instead of decision trees

This seems to be pretty standard. With logistic regression as the base estimator, the adaptive boosting stops adding value after very few iterations. I put together a little notebook to illustrate, ...
• 4,723

"Feature selection", as you describe it, boils down to fitting a depth 1 decision tree on a set of data. Given such a set and weights associated with each, the problem of fitting a decision tree (a ...
Accepted

The "M" stands for "Multi-class" (Hi Ébe! I'm a little late to the party.), in contrast to binary, i.e., two-class. For example, Freund and Schapire have "First Multi-class Extension" as the title ...
• 56
Accepted

Fitting to a weighted sample works the same way for pretty much any statistical/machine learning model: you minimize the sample weighted loss function. In the case of logistic regression, you would ...
Accepted

The choice to use the weighted median appears to be arbitrary. According to this "the idea of using the weighted median as the final regressor is not new. Freund [6] briefly mentions it and proves a ...
• 1,347
Accepted

You can do something similar, mathematically, but with a slightly different sigmoidal function to what you specified. To convert the output of AdaBoost $$F(x) = \sum_{t=1}^T \alpha_t h_t (x)$$ to a ...
• 2,151

### How to ensure that increasing the weights of misclassified points in AdaBoost does not adversely affect the learning progress?

The new classifier in each round might indeed classify the old points incorrectly. However the previous 'versions' of the classifier(from previous iterations) are not thrown away. The end result is an ...
• 1,465
Accepted

The reason for using 'stumps' in boosting but full-height trees in random forests is to do with how the aggregation and fitting is done. In random forests, the trees in the ensemble are fitted ...
• 40.3k
Accepted

### Boosting reduces bias when compared to what algorithm?

It is said that bagging reduces variance and boosting reduces bias. Indeed, as opposed to the base learners both ensembling methods employ. For bagging and random forests, deep/large trees are ...
• 2,676
Accepted

### Using Bayes for combining forecasts with different accuracies (Interview question)

With probabilistic forecasts, accuracy cannot be simply defined as (number correct forecasts)/(total number of forecasts), or better, you must define what you mean by a "correct forecast". For example ...
• 18.1k
Accepted

### Why AdaBoost works exactly the way it does

I will justify the whole algorithm from the statistical point of view, i.e. risk minimization. I will get to the weights later. Now, just for the record, I will outline steps of the AdaBoost algorithm....
• 542
Accepted

As Lucas already said (+1), yes, your intuition is correct; that is the sum of the weighted error terms. This steps is indeed awkward for Adaboost, because it is ...
• 44.7k
Accepted

### How do you interpret prediction output in GBM() in R for classification problem?

According to gbm's reference manual: While indeed type="response" then gbm converts back to ...
• 44.7k
Accepted

### What's the purpose of learning rate in sklearn AdaBoost implementation

The learning rate is just applied to each of the tree's predictions and has nothing to do with the tree model itself but the boosting 'meta' algorithm. Since boosting is iteratively learning from the ...
• 1,562