Boosting is a process of finding & combining weakly predictive models into a strongly predictive model.

learn more… | top users | synonyms

0
votes
0answers
9 views

How many iterations we have to perform in adaboost classification?

How many iterations we have to perform in adaboost classification? As the number of iteration increases error rate gradually reduces and sometimes classification accuracy goes upto 100% in both ...
5
votes
1answer
74 views

Where must we use Bagging or Boosting?

I want to know when Bagging is better than Boosting? How I select appropriate method for my classification task? I think when we have many outliers in our data-set, Bagging must be better than ...
0
votes
1answer
70 views

Reconciling boosted regression trees (BRT), generalized boosted models (GBM), and gradient boosting machine (GBM)

Questions: What is the difference(s) between boosted regression trees (BRT) and generalized boosted models (GBM)? Can they be used interchangeably? Is one a specific form of the other? Why did ...
2
votes
1answer
51 views

Random forest vs Adaboost

In section 7 of the paper Random Forests (Breiman, 1999), the author states the following conjecture: "Adaboost is a Random Forest". Has anyone proved, or disproved this? What has been done to prove ...
0
votes
0answers
59 views

Xgboost Regresion tree

I am building a boosted regression tree in R and I use the simple xgboost function from the package xgboost in R. ...
2
votes
1answer
53 views

Meaning of the Boosting algorithm for Regression Trees

I have a problem with understanding the concept of the Boosting Algorithm. ...
1
vote
1answer
37 views

out-of-bag error estimate for Boosted Trees

In Random Forest, each tree is grown in parallel on a unique boostrap sample of the data. Because each boostrap sample is expected to contain about 63% of unique observations, this lefts roughly 37% ...
1
vote
0answers
33 views

Huge overfitting with Random Forests and Boosted Trees?

In the following picture, the boxplots represent a performance metric (the closer to 1, the better) recorded for 50 runs of cross-validation, and the black filled circles are the training values of ...
3
votes
2answers
78 views

Are Random Forests and Boosting parametric or non-parametric?

From this excellent paper by Breiman, we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forests, ...
1
vote
0answers
38 views

Boosting Explained

I'm a newbie trying to learn Boosting. The examples I found online are quite confusing. Is there a simple tutorial somewhere that explains ...
3
votes
0answers
27 views

GradientBoostClassifier(sklearn) takes very long time to train

I'm using dataset with 61879 datapoints and 102 features. On this dataset Randomforest(sklearn) takes less than 90s to train for 100 estimators while GradientBoostClassifier(sklearn) is taking forever ...
1
vote
1answer
30 views

Skewed Classification Problem

So I've read around and seen this is a problem. I have a classification problem and 12 variables ... I'm working on getting more, but even if l get the number to 20-30 I feel like the problem will ...
0
votes
1answer
55 views

Meaning of `max_depth` in GradientBoostingClassifier in scikit-learn

when I use the GradientBoostingClassifier from scikit-learn, I find that there is a parameter max_depth to set, which controls the maximum depth of the regression ...
0
votes
1answer
25 views

How to choose a regression tree (base learner) at each iteration of Gradient Tree Boosting?

I'm trying to understand Gradient Tree Boosting, by following Prof. Friedman's original paper: Greedy Function Approximation: A Gradient Boosting Machine. Basically, at each iteration, a regression ...
1
vote
1answer
26 views

Why does the equation $ -\sum^{n}_{t=1} \tilde{W}(t)_{m-1} y_{t}h(x; \theta_{m}) = 2 \epsilon_m -1$ hold in boosting?

I was trying to understand the boosting algorithm as described by the MIT graduate class lectures notes on ocw. On page 2 they give the outline of boosting as follows: The step that is not clear ...
1
vote
1answer
93 views

Boosted trees and Variable Interactions

How can one see in a Boosted trees classification model, which variables interact with each other and how much? I would like to make use o this in R gbm package if possible
0
votes
0answers
50 views

Value of the loss function and Classification Errors in gbm package (R)

I have a simple problem of classification (0s and 1s) using adaboost loss function. When I check the components of a boosted model using the gbm package I see: ...
0
votes
0answers
40 views

why boosting method is sensitive to outliers

I found many articles showing that boosting methods are sensitive to outliers, but no article explains why. In my experience, I feel outliers data is bad for any machine learning algorithms, but why ...
1
vote
1answer
28 views

Alternative to AIC for feature selection in classification

I want to know what are the most common methods for feature selection in classification problems (binary and mutli-class). I see in Chapter 6 of Zumel and Mount that they use AIC before they train ...
0
votes
0answers
47 views

what is the difference between bagging and boosting in random forest?

I understand what is bagging and how it is applied to random forest. But how is bagging different from boosting. If boosting is different from bagging, how can boosting be applied to random forest?
2
votes
1answer
63 views

How are individual trees added together in boosted regression tree?

I'm reading Introduction to Statistical Learning, James, G., et al. (2013), in which they describe the Boosted Regression Tree algorithm as following. What I do not understand is Eq 8.10 and 8.11. ...
0
votes
0answers
111 views

Best machine learning methtod for classificating datasets with non-independent cases within the groups

I have to perform binary classification of my data with supervised machine learning, but I have some difficulties working with my data set. It consists many genetic mutations that have parameters ...
1
vote
1answer
67 views

AdaBoost over blackbox weak classifier

Can I somehow implement AdaBoost procedure over a weak classifier from another library? For example over SVM from libsvm, or over some neural network. The idea of AdaBoost is that current weights of ...
12
votes
0answers
158 views

Prediction interval based on cross-validation (CV)

In the text books and youtube lectures I learned a lot about iterative models such as boosting, but I never saw anything about deriving a prediction interval. Cross validation is used for the ...
0
votes
0answers
91 views

Combining multiple feature subsets through ensemble classification methods?

I have a set of $N$ samples to be classifies in a binary classification problem. I have extracted features from these samples from 4 different perspectives (views) of every samples. Hence I have 4 ...
1
vote
0answers
38 views

What is “fitted function” in the context of boosted regression tree?

I'm following the tutorial of package dismo's boosted regression tree, which produces two graphs, about fitted function and ...
1
vote
2answers
22 views

Incremental improvement for boosting

By adding additional factors, will the fitting result of a boosting algo (say Ada boosting) guaranteed to be improved? From my experiment, adding additional factors could make the prediction accuracy ...
1
vote
1answer
87 views

Posterior probabilities with decision trees or decision forests

Is there a way to get posterior probabilities $P(C | \vec{x})$ (probability that a data item $\vec{x}$ belong to one of the given classes) in a multiclass classification problem using decision trees ...
0
votes
1answer
17 views

For the given type of dataset, what would generally be the set of classifiers that should be tried to get the highest TPR for FPR = 0.01

I'm primarily looking to attain the maximum True Positive Rate for a small False positive Rate of say 0.01. The following is an instance: 1 37.33 228.39 0 77.060599 0.073384 0.052536 ...
1
vote
2answers
399 views

Why Adaboost with Decision Trees?

I've been reading a bit on boosting algorithms for classification tasks and Adaboost in particular. I understand that the purpose of Adaboost is to take several "weak learners" and, through a set of ...
0
votes
1answer
115 views

How to choose an appropriate maxdepth in rpart.conrol?

I'm using the boosting method in adabag library and trying to choose an appropriate maxdepth in rpart.control for building a 2-class classification model using my training dataset. I have noticed that ...
1
vote
0answers
83 views

How can I compare GBM feature importances to GBM partial dependence plots?

I am having trouble reconciling the difference between the indicated "importance" from a GBM that I am calculating with what is shown in the partial dependence plots. I would expect higher ...
1
vote
0answers
67 views

Different variable importance results with stabsel and mboost

I'm using glmboost in the mboost package to fit a boosted regression using linear models as the base learner. There are 13200 observations and about 75 variables, ...
0
votes
0answers
83 views

Parameter selection for GBM

I'm building a Gradient Boosting model. Given a dataset and event rate, is it possible to get a formula/ definitive strategy for the optimum number of trees, shrinkage parameter and depth of trees? I ...
0
votes
0answers
20 views

Combining features extracted from different parts of the same image

It is about car identification in images. I have an 64x64 image divided into 16 equal windows. I compute a HoG features algorithm in each one. And I am using the concatenation vector resulted from ...
0
votes
0answers
18 views

Would knowing the underlying distribution for our data affect how boosting searches for its predictor or how it minimizes the exponential loss?

Assume that the goal of Machine learning is to find a function that is able to minimize the generalization/expected/true error (assuming that the underlying distribution is fixed but unknown): $$E(f) ...
2
votes
0answers
141 views

Is multicollinearity a problem with gradient boosted trees (i.e. GBM)?

A question about multicollinearity for random forests has been asked and answered, but what about boosted trees?
2
votes
1answer
164 views

Has anyone publicly shared an implementation of RUSBoost in R?

There's no package available on CRAN, so I was hoping someone in the community had written their own function/package. I see it's been done in MATLAB, so I may just have to start with that and write ...
1
vote
0answers
35 views

Classifiers that support weighting of the instances

I'm thinking of implementing my own boosting algorithm, so I'm looking for any multiclass classfication algorithms that would support weighting of the examples, i.e. you could specify what examples ...
0
votes
0answers
54 views

Adaboost for neural networks. Is it still worth it?

I have a question about Adaboost and neural networks. Given the recent development in neural networks (dropout, maxout, or rectified linear units) is there a significant benefit of performing Adaboost ...
1
vote
0answers
156 views

Adaptive Boosting vs. SVM

I am working on a binary classification case and comparing the performance of different classifiers.Testing the performance of adaboost algorithm (with decision ...
3
votes
1answer
300 views

Why not always use ensemble learning?

It seems to me that ensemble learning WILL always give better predictive performance than with just a single learning hypothesis. So, why don't we use them all the time? My guess is because of ...
3
votes
1answer
156 views

Why should all Cross-Validation results be higher than the result on the test dataset?

Sorry, I'm not an expert and my question could be fundamentally wrong. I've read this interesting question because I also was wondering whether to train the model again after cross-validation. Now, ...
0
votes
0answers
51 views

How to extract the importance of predictors in Adaptive boosting?

I have trained a boosting classifier using ada package in R. Now I want to see the importance of my predictors in constructing ...
0
votes
0answers
64 views

boosting with ada package. How do I take the most probable answer from predict?

I trained classifier using ada. Now I executed: predict(adaDol,newdata=cords()) and received response: ...
0
votes
0answers
15 views

Is it possible that boosting doesn't increase the predictive power of tree?

I have a data set of 282 observations, and my response variable is a binary variable where 0=normal and 1 =disease. I constructed a classification tree with rpart ...
0
votes
0answers
83 views

Adaboost and factor variables

My dataset contains both numerical and categorical features like education level, region etc (i use factor variable type for them). I think these variables are important for predicting the outcome of ...
0
votes
0answers
114 views

BRT analysis using count data

I have some problems with my BRT analysis. Introduction to the data: The dependent variable is count data of a specific palm species in SA, and the predictors consists of nine various kinds of ...
3
votes
2answers
526 views

How to find optimal values for the tuning parameters in boosting trees ?

I realise that there are 3 tuning parameters in the boosting trees model, i.e. the number of trees (number of iterations) shrinkage parameter number of splits (size of each constituent trees) My ...
1
vote
1answer
134 views

How is L2 Boosting Different from a Big Regression Tree?

I'm learning about boosting. I think I understand how adaptive boosting works for classification. I'm trying to get some intuition for regression boosting. At each iteration, adaptive boosting ...