Boosting is a process of finding & combining weakly predictive models into a strongly predictive model.

learn more… | top users | synonyms

4
votes
1answer
66 views

What does it mean to “fit a regression function” and then use it to update other functions?

Referring to the algorithm on page 11 in this paper on boosting algorithms, I really don't understand step 2, (ii) and (iii). What does this mean: (ii) Fit the regression function $g_j^h (x)$ by ...
0
votes
0answers
3 views

Boosting in Weka with single PC

Given that I want to run Weka in a single PC, is it a good a idea to use boosting with a three layer feedforward neural network as a base learner? Why or why not? I think boosting requires a lot of ...
1
vote
0answers
8 views

BRT predictions on zero-inflated gaussian fish abundances include negative results

hopefully someone can point me in the right direction here. I'm using boosted regression trees (BRT) to assess the relative importance of a number of environmental factors (sea bottom temperature, ...
1
vote
0answers
41 views

R: What do I see in partial dependence plots of gbm and RandomForest?

Actually, I thought I had understood what one can show a with partial dependence plot, but using a very simple hypothetical example, I got rather puzzled. In the following chunk of code I generate ...
1
vote
0answers
17 views

weighted conditional expectation in adaboost

I'm looking at this paper, http://www.stanford.edu/~hastie/Papers/AdditiveLogisticRegression/alr.pdf around pages 10-11 (marked 346,347 in the pdf). This notation is introduced $$ E_w[g(x,y)|x] := ...
0
votes
0answers
32 views

Adaboost: weak learn on different dataset for each iterations

I have seen a few occasion were a weak learner was used iteratively with different dataset. For example, let's say you want to learn to classify {1, -1} according to 3 properties and naive Bayes weak ...
0
votes
1answer
24 views

Applying ensemble learning to quantile regression?

Is it desirable / possible to apply ensemble learning methods (boosting, bagging, etc) to the quantile regression problem?
0
votes
0answers
14 views

what does Task in multitask learning mean?

I'm new in multitask learning. I didn't exactly understand what is Tasks in multitask learning ? every task are single training data or what ?
0
votes
1answer
42 views

Shrinkage parameter in Adaboost?

I'm unclear how the shrinkage parameter works in Adaboost. I understand the concept of shrinkage in the theoretical sense related to ordinary least squares, but I'm not sure how to interpret this ...
0
votes
0answers
26 views

Default prediction threshold for two-class gradient boosting classification

At the bottom of page 9 in http://statweb.stanford.edu/~jhf/ftp/trebst.pdf, the author relates the final function $F_{m}$ to $P(y=1|x)$ via the logistic function, and suggests classifying a data point ...
1
vote
2answers
95 views

How Gradient boosting can be more interpretable than CART?

I found this document which compare some learning methods and I don't understand this table : Gradient boosting has a better intepratability score than CART. How is it possible ? I thought gradient ...
1
vote
0answers
53 views

Gradient boosting algorithm (steps) question

So, far I have read following regarding boosting: Boosting is an ensemble technique. Train learner sequentially, where early learners fit simple models to the data. Analyze data for errors, that ...
0
votes
1answer
59 views

C5.0, boosting, and mislabeled data

I'm trying to model a binary classification problem. 5 continuous features, slightly imbalanced dataset (about 60 in one class, a little more than 200 in another). So far I've tried kNN, LDA, and ...
6
votes
4answers
312 views

Random Forest, is it a boosting algorithm?

Short definition of boosting: Can a set of weak learners create a single strong learner? A weak learner is defined to be a classifier which is only slightly correlated with the true ...
0
votes
1answer
125 views

Gradient boosting in R uses only a single variable

I am trying to build a boosting model using the package gbm in R. I have the following code: ...
3
votes
1answer
74 views

Multiple definitions of AdaBoost

The description of AdaBoost in Kevin Murphy's Machine Learning book (shown in a snapshot below) differs from the one in Wikipedia. I am trying to relate both definitions. Step by step, my questions ...
0
votes
1answer
109 views

boosting with linear svm

I am working on boosting classifier. I am planning to use linear svm as the weak classifier. I am using liblinear for it. My question is how can I weight each instance of liblinear based on the ...
0
votes
0answers
30 views

Minimizing Specificity in Adaboost (and other) Classifiers

This is a relatively simple question, but is not addressed in the documentation for the ada package in R. How do I go about instructing ...
0
votes
0answers
220 views

C5.0 Node Decision Tree with Boosting Algorithm (SPSS Modeler)

We are currently looking to mimic the SPSS Modeler's C5.0 Node decision tree model with boosting, which means we'd like to rewrite the whole thing in plain Java. I ...
0
votes
1answer
56 views

componentwise boosting based on fisher scoring

componentwise boosting dates back at least to Bühlman and Yu (2003), where in each boosting iteration a set of base-learners (e.g. simple linear models) depending on a subset of the covariates are ...
0
votes
0answers
111 views

R: How do boosted regression trees deal with missing data?

How does the R implementation of boosted regression trees (package gbm) by default deal with missing values of the predictor variables? Are they imputed and if they are, according to which algorithm? ...
0
votes
0answers
115 views

Weak Classifiers weights/contribution in Adaboost and Real adaboost?

In Adaboost according to SAMME implementation, the $\alpha$ determines the contribution of the weak classifier. Here is the Adaboost algorithm $\alpha$ is in step 2. (c) Now in RealAda boost I ...
1
vote
0answers
42 views

IRLS working weights proportional odds

According to Reduced-rank vector generalized linear models the parameter estimates are obtained by Fisher-scoring algorithm. In the VGAM package the IRLS algorithm ...
2
votes
1answer
104 views

Boosting algorithms and overfitting

Reading the book of Witten and Frank on Data Mining alogorithms i read: "If boosting succeeds in reducing the error on fresh test data, it often does so in a spectacular way. One very surprising ...
0
votes
0answers
85 views

Probabilistic outputs from LogitBoost

I have some code to train an AdaBoost model using decision stumps. I converted the outputs into probabilities via logistic correction, as described here. Specifically, if the boosted model outputs ...
3
votes
1answer
118 views

Why is boosting effective?

Boosting takes a bunch of weak learners and creates a strong learner. But why is it so difficult to create a strong learner right from the beginning without using boosting techniques? And therefore ...
2
votes
1answer
115 views

Is there a well-defined class of ensemble methods?

Ensemble methodology's main aim is to somehow aggregate or summarize estimates from multiple models. In some cases this is aggregating different bootstrap estimates or Monte Carlo estimates, but ...
1
vote
0answers
82 views

GBDT and model building: How am I overfitting?

Here's my situation: Binary classification and I've got a training set of roughly 250k samples and 10 features, and a validation set of roughly 100k with the same number of features. I'm fitting GBDT ...
1
vote
0answers
38 views

Adaboosted trees and q-learning

Is there a solid mathematical connection between ADABOOST and Q-learning that can inform "higher performance" machine learning algorithms? Q-learning references: ...
1
vote
1answer
124 views

The weight updating in adaboost

1.AdaBoost updates the weight of the sample By the current weak classifier in training each stage. Why doesn't it use the all of the previous weak classifiers to update the weight. (I had tested it ...
2
votes
1answer
50 views

How can I do logistic correction for boosting

Can anyone tell me if logistic correction is the best method to correct the probability of gradient boosting machine? If so, how can I do it?
1
vote
0answers
74 views

Which Regression methods are suitable for binary valued features and continuous output?

I want to build a machine learning model to regression on continuous output given binary valued features(0,1). the dimension of my problem is around 200. which of the flowing methods seems suitable ...
1
vote
4answers
181 views

Measuring representativeness of a sample using covariates

I was provided with quite a small sample of labeled (variable of interest) observations to train a model to predict unlabeled observations. All the observations are associated with many covariates. ...
0
votes
0answers
207 views

Obtaining final classification score using AdaBoost predict function

If I understand correctly, predict.ada() returns an $n$ by 2 matrix of class probabilities for each classifier used in $n$ iterations. How can I obtain the final classification on scale of [0,1] for ...
2
votes
1answer
105 views

probability distribution of output value with regression tree methods

If I have a regression problem where I try to estimate the value of $y$ as function of $x_1 \dots x_d$: $$ y = f(x_1,\dots,x_d) $$ using a Boosted Regression Tree or a Random Forest Regression, is it ...
3
votes
2answers
644 views

Random forests vs boosting

I thought it would be interesting to talk about two of the best ensemble methods off-the-shelf: Random Forests and Boosting. When would you apply one method rather than the other one?
3
votes
1answer
491 views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ ...
2
votes
0answers
84 views

Is there a theoretical basis for the shrinkage used in Boosted Regression Trees?

In Gradient Boosted Regression Trees, a shrinkage $\nu$ is often applied as: $$ f_t(x) \leftarrow f_{t-1}(x) + \nu h(x)$$ where $h$ is the regression tree learned by fitting the tree to the gradient. ...
2
votes
1answer
164 views

How to combine a SVM classifier and a Naive Bayes classifier

I have two different set of features for which I have a SVM classifier and a Naive Bayes classifier, respectively. If I wanted to combine these two classifiers to get a better prediction, what option ...
2
votes
1answer
394 views

Interaction depth parameter in GBM

In the GBM package one is supposed to be able to provide interaction.depth>2, which means higher-order interactions between features. However, the resulting trees (as seen by pretty.gbm.tree) never ...
4
votes
1answer
115 views

Regarding the sampling procedure in Adaboost algorithm

The AdaBoost algorithm states that it is to train a classifier based on the training data according to a weight vector. Assume the size of training data is N, the weight vector is of dimension N as ...
3
votes
0answers
41 views

Tutorials / examples for multiclass boosting

I want to learn the multiclass boosting technique. I have a basic understanding of binary boosting and also have seen some working examples on this. I have also read about the basics of multiclass ...
3
votes
1answer
343 views

How to use CART for AdaBoost?

I am trying to use CARTs (Classification and Regression Trees) for AdaBoost as weak learner. My question concerns the update of the weights after fitting the best weak learner. A single CART node ...
1
vote
0answers
179 views

Adaboost feature weight calculation

I thought I understood Adaboost, until code analysis made me realize that sample_weight is not an array of the feature weights... and after further investigation I am left confused as to how ...
2
votes
1answer
2k views

What is the difference between AdaBoost.M2, AdaBoost.M1, Gentle AdaBoost, RealAdaboost and the Original Adaboost?

I know some questions have been made for example between gentle and ada, but I was wondering about gentle and the others. For instance, I saw two different works talking about AdaBoost.M1 and ...
2
votes
0answers
102 views

How to modify RankBoost to maximize area under recall-precision curve instead of AUC?

Using the WeakLearn algorithm from the original RankBoost paper, how do you set the optimal threshold to maximize AU-RPC (instead of AUC)? And, once that threshold is set, how do you calculate the ...
0
votes
1answer
139 views

How do I do multiscale HMM classification?

I'm using hidden Markov models to classify some accelerometer data. I take the Fourier transform of the raw data at a given window length, and then train an HMM for each class, and every test instance ...
5
votes
1answer
501 views

Common weak learners for Adaboost

I'm looking for a set of weak classifiers that work with Adaboost to test on popular datasets. Most of the examples on the web use some kind of random weak learners which work on their own randomly ...
2
votes
1answer
265 views

Concept of iterations in Adaboost

I can't seem to get my head around "iterations" in Adaboost. Are they analogous to weak classifiers that are used for Boosting? I've seen many examples of Adaboost where a programmers use a Single ...
0
votes
1answer
124 views

Accuracy of classifiers with Adaboost

Does Adaboost ensure that resultant accuracy is more than or at least equal to current accuracies? What happens if Classifier A performs badly and the weights are accordingly updated and the next ...