2
votes
2answers
87 views

How to find optimal values for the tuning parameters in boosting trees ?

I realise that there are 3 tuning parameters in the boosting trees model, i.e. the number of trees (number of iterations) shrinkage parameter number of splits (size of each constituent trees) My ...
1
vote
0answers
46 views

How is L2 Boosting Different from a Big Regression Tree?

I'm learning about boosting. I think I understand how adaptive boosting works for classification. I'm trying to get some intuition for regression boosting. At each iteration, adaptive boosting ...
0
votes
0answers
16 views

how to generate samples for model selection testing on adaboost

I want to assess some statistical methods for model selection on binary classification using adaBoost. For this, I have to generate artificial samples (input data) and create an oracle that has the ...
0
votes
0answers
14 views

what does Task in multitask learning mean?

I'm new in multitask learning. I didn't exactly understand what is Tasks in multitask learning ? every task are single training data or what ?
0
votes
1answer
68 views

Shrinkage parameter in Adaboost?

I'm unclear how the shrinkage parameter works in Adaboost. I understand the concept of shrinkage in the theoretical sense related to ordinary least squares, but I'm not sure how to interpret this ...
7
votes
4answers
592 views

Random Forest, is it a boosting algorithm?

Short definition of boosting: Can a set of weak learners create a single strong learner? A weak learner is defined to be a classifier which is only slightly correlated with the true ...
0
votes
1answer
159 views

Gradient boosting in R uses only a single variable

I am trying to build a boosting model using the package gbm in R. I have the following code: ...
0
votes
1answer
216 views

boosting with linear svm

I am working on boosting classifier. I am planning to use linear svm as the weak classifier. I am using liblinear for it. My question is how can I weight each instance of liblinear based on the ...
0
votes
0answers
41 views

Minimizing Specificity in Adaboost (and other) Classifiers

This is a relatively simple question, but is not addressed in the documentation for the ada package in R. How do I go about instructing ...
0
votes
1answer
59 views

componentwise boosting based on fisher scoring

componentwise boosting dates back at least to Bühlman and Yu (2003), where in each boosting iteration a set of base-learners (e.g. simple linear models) depending on a subset of the covariates are ...
0
votes
0answers
145 views

Weak Classifiers weights/contribution in Adaboost and Real adaboost?

In Adaboost according to SAMME implementation, the $\alpha$ determines the contribution of the weak classifier. Here is the Adaboost algorithm $\alpha$ is in step 2. (c) Now in RealAda boost I ...
3
votes
1answer
120 views

Why is boosting effective?

Boosting takes a bunch of weak learners and creates a strong learner. But why is it so difficult to create a strong learner right from the beginning without using boosting techniques? And therefore ...
1
vote
0answers
40 views

Adaboosted trees and q-learning

Is there a solid mathematical connection between ADABOOST and Q-learning that can inform "higher performance" machine learning algorithms? Q-learning references: ...
1
vote
1answer
139 views

The weight updating in adaboost

1.AdaBoost updates the weight of the sample By the current weak classifier in training each stage. Why doesn't it use the all of the previous weak classifiers to update the weight. (I had tested it ...
1
vote
0answers
81 views

Which Regression methods are suitable for binary valued features and continuous output?

I want to build a machine learning model to regression on continuous output given binary valued features(0,1). the dimension of my problem is around 200. which of the flowing methods seems suitable ...
1
vote
4answers
183 views

Measuring representativeness of a sample using covariates

I was provided with quite a small sample of labeled (variable of interest) observations to train a model to predict unlabeled observations. All the observations are associated with many covariates. ...
3
votes
1answer
131 views

Is there a theoretical basis for the shrinkage used in Boosted Regression Trees?

In Gradient Boosted Regression Trees, a shrinkage $\nu$ is often applied as: $$ f_t(x) \leftarrow f_{t-1}(x) + \nu h(x)$$ where $h$ is the regression tree learned by fitting the tree to the gradient. ...
4
votes
1answer
135 views

Regarding the sampling procedure in Adaboost algorithm

The AdaBoost algorithm states that it is to train a classifier based on the training data according to a weight vector. Assume the size of training data is N, the weight vector is of dimension N as ...
2
votes
0answers
121 views

How to modify RankBoost to maximize area under recall-precision curve instead of AUC?

Using the WeakLearn algorithm from the original RankBoost paper, how do you set the optimal threshold to maximize AU-RPC (instead of AUC)? And, once that threshold is set, how do you calculate the ...
5
votes
1answer
603 views

Common weak learners for Adaboost

I'm looking for a set of weak classifiers that work with Adaboost to test on popular datasets. Most of the examples on the web use some kind of random weak learners which work on their own randomly ...
2
votes
1answer
450 views

How to determine whether a classifier like adaboost is weak?

I run the cross-validation experiment for a given data set, and tried two different approaches: one is based on SVM, another is based on SVM plus Adaboost. But the confusion matrix for two experiments ...
7
votes
2answers
768 views

Classification with GBM in R and imbalanced class sizes

I'm dealing with a supervised binary classification issue. I'd like to use the GBM package to classify individuals as uninfected/infected. I have 15 times more uninfected than infected individuals. I ...
2
votes
1answer
619 views

Regarding boosting, bagging and bootstrapping [closed]

How to understand the relationships, comparative advantages, and comparative disadvantages of boosting, bootstrapping and bagging in terms of their respective applications in data mining.
3
votes
1answer
397 views

Selecting features using Adaboost

How does Adaboost select best features from the sample data (or a unit feature vector)? It would be nice if someone can explain if the above statement is true or not. I've seen the term features and ...
5
votes
2answers
350 views

Base classifiers for boosting

Boosting algorithms, such as AdaBoost, combine multiple 'weak' classifiers to form a single stronger classifier. Although in theory boosting should be possible with any base classifier, in practice it ...
4
votes
1answer
120 views

Continuous Multistate Ada-boost method?

I recently read this simple introduction to Adaboost as a review after learning about it a few years ago. This is in preparation for actually using it to solve a problem I am working on which is ...
11
votes
2answers
1k views

In boosting, why are the learners “weak”?

See also a similar question on stats.SE. In boosting algorithms such as AdaBoost and LPBoost it is known that the "weak" learners to be combined only have to perform better than chance to be useful, ...
6
votes
1answer
1k views

Is AdaBoost less or more prone to overfitting?

I have read various (seemingly) contradicting statements whether or not AdaBoost (or other boosting techniques) are less or more prone to overfitting compared to other learning methods. Are there ...
1
vote
0answers
276 views

Obtaining resampling based estimates of prediction error in boosted regression tree model

I try to use the gbm.fit() function for a boosted regression tree model implemented in the R package gbm. To investigate e.g., the bootstrapped prediction error and ...
6
votes
1answer
339 views

Understanding similarity sensitive hashing algorithm in AdaBoost

I'm a CS major and don't quite understand the mathematics behind a optimization problem coming from a machine learning algorithm. The algorithm is in Section 5 of the paper ...
3
votes
2answers
316 views

What to do when weak classifiers are almost identical in AdaBoost?

I have written some code which uses AdaBoost to generate a set of weak classifiers. I'm finding, however, that when I use the resulting strong classifier to classify examples, it seems that almost ...
10
votes
2answers
2k views

What does interaction depth mean in GBM?

I had a question on the interaction depth parameter in gbm in R. This may be a noob question, for which I apologize, but how does the parameter, which I believe denotes the number of terminal nodes in ...
2
votes
1answer
182 views

Different optimal number of boosting iterations obtained from OOB and on test

If I'm using a machine learning model (e.g. boosted regression trees like gbm in R) on a dataset, what does it mean if there's a significant difference between the OOB estimated optimal # of ...
13
votes
2answers
709 views

On the “strength” of weak learners

I have several closely-related questions regarding weak learners in ensemble learning (e.g. boosting). This may sound dumb, but what are the benefits of using weak as opposed to strong learners? ...
3
votes
1answer
176 views

AdaBoost on a continuum of base classifiers

A tutorial on AdaBoost suggests that AdaBoost can be applied to a continuum of classifiers (at the bottom of the first page). Does it mean to simply discretize the classifiers, for example, which are ...
2
votes
1answer
314 views

The upper bound of the training error of AdaBoost

I am reading an overview of AdaBoost written by Schapire, which calculates the upper bound of the training error in Eq. (5), section 3. In fact, it states that ...
2
votes
1answer
450 views

How to choose the 1st threshold/classifier/ weak learner in Adaboost?

I am having some difficulty understanding Adaboost. How should the 1st threshold/classifier/weak learner be chosen? It seems that there are two conditions which must be satisfied Choose the ...
5
votes
1answer
1k views

When would one want to use AdaBoost?

As I've heard of the AdaBoost classifier repeatedly mentioned at work, I wanted to get a better feel for how it works and when one might want to use it. I've gone ahead and read a number of papers and ...
3
votes
1answer
1k views

Adjusting sample weights in AdaBoost

I am trying to read up about AdaBoost from Tibshirani (page 337 onwards), and would appreciate some help in understanding it better. The book says that "For each successive iteration m = 2, 3, . . ...
8
votes
1answer
889 views

Calibrating a multi-class boosted classifier

I have read Alexandru Niculescu-Mizil and Rich Caruana's paper "Obtaining Calibrated Probabilities from Boosting" and the discussion in this thread. However, I am still having trouble understanding ...
6
votes
1answer
410 views

When is the shrinkage applied in Friedman's stochastic gradient boosting machine?

In boosting, each additional tree is fitted to the unexplained variation in the response that is currently un-modelled. If we are using squared-error loss, this amounts to fitting on the residuals ...
10
votes
1answer
2k views

How does gentle boosting differ from AdaBoost?

There is a variant of boosting called gentleboost. How does gentle boosting differ from the better-known AdaBoost?
14
votes
3answers
1k views

How does boosting work?

What is the easiest way to understand boosting? Why doesn't it boost very weak classifiers "to infinity" (perfection)?