214 votes

Gradient Boosting Tree vs Random Forest

$\text{error = bias + variance}$ Boosting is based on weak learners (high bias, low variance). In terms of decision trees, weak learners are shallow trees, sometimes even as small as decision stumps (...
user avatar
  • 5,820
91 votes

Bagging, boosting and stacking in machine learning

Bagging: parallel ensemble: each model is built independently aim to decrease variance, not bias suitable for high variance low bias models (complex models) an example of a tree based method is ...
user avatar
  • 1,254
69 votes

Gradient Boosting Tree vs Random Forest

This question is addressed in this very nice post. Please take a look at it and the references therein. http://fastml.com/what-is-better-gradient-boosted-trees-or-random-forest/ Notice in the article ...
user avatar
  • 13.1k
58 votes

Bagging, boosting and stacking in machine learning

Just to elaborate on Yuqian's answer a bit. The idea behind bagging is that when you OVERFIT with a nonparametric regression method (usually regression or classification trees, but can be just about ...
user avatar
  • 2,239
56 votes
Accepted

US Election results 2016: What went wrong with prediction models?

In short, polling is not always easy. This election may have been the hardest. Any time we are trying to do statistical inference, a fundamental question is whether our sample is a good ...
user avatar
  • 17.9k
46 votes

Won't highly-correlated variables in random forest distort accuracy and feature-selection?

Old thread, but I don't agree with a blanket statement that collinearity is not an issue with random forest models. When the dataset has two (or more) correlated features, then from the point of view ...
user avatar
  • 561
45 votes
Accepted

Gradient Boosting for Linear Regression - why does it not work?

What am I missing here? I don't think you're really missing anything! Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as ...
user avatar
39 votes
Accepted

Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

Actually, scikit-learn does provide such a functionality, though it might be a bit tricky to implement. Here is a complete working example of such an average ...
user avatar
  • 506
36 votes

US Election results 2016: What went wrong with prediction models?

There are a number of sources of polling error: You find some people hard to reach This is corrected by doing demographic analysis, then correcting for your sampling bias. If your demographic ...
user avatar
  • 610
31 votes

US Election results 2016: What went wrong with prediction models?

This was mentioned in the comments on the accepted answer (hat-tip to Mehrdad), but I think it should be emphasized. 538 actually did this quite well this cycle*. 538 is a polling aggregator that ...
user avatar
  • 411
30 votes
Accepted

k-fold Cross validation of ensemble learning

Ensemble learning refers to quite a few different methods. Boosting and bagging are probably the two most common ones. It seems that you are attempting to implement an ensemble learning method called ...
user avatar
  • 636
30 votes
Accepted

Won't highly-correlated variables in random forest distort accuracy and feature-selection?

That is correct, but therefore in most of those sub-samplings where variable Y was available it would produce the best possible split. You may try to increase mtry, to make sure this happens more ...
user avatar
29 votes
Accepted

How do ensemble methods outperform all their constituents?

It's not guaranteed. As you say, the ensemble could be worse than the individual models. For example, taking the average of the true model and a bad model would give a fairly bad model. The average ...
user avatar
  • 9,655
24 votes

Bagging, boosting and stacking in machine learning

See my ensemble learning blog post Sources for this image: Wikipedia sklearn
user avatar
  • 1,499
21 votes

What is the difference between bagging and random forest if only one explanatory variable is used?

The fundamental difference is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, ...
user avatar
21 votes
Accepted

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

As noticed in the comments, you’ve re-discovered boosting. Nothing wrong with this approach, but usually it’s easier and safer to use a method already implemented and battle-tested by someone else ...
user avatar
  • 113k
19 votes
Accepted

hard voting, soft voting in ensemble based methods

Let's take a simple example to illustrate how both approaches work. Imagine that you have 3 classifiers (1, 2, 3) and two classes (A, B), and after training you are predicting the class of a single ...
user avatar
  • 11.9k
17 votes

When should I not use an ensemble classifier?

I do not recommend using an ensemble classifier when your model needs to be interpretable and explainable. Sometimes you need predictions and explanations of the predictions. When you need to ...
user avatar
  • 341
17 votes

US Election results 2016: What went wrong with prediction models?

First it was Brexit, now the US election Not really a first, e.g. the French presidential election, 2002 "led to serious discussions about polling techniques". So it's not far-fetched to say these ...
user avatar
17 votes
Accepted

How does gradient boosting calculate probability estimates?

TL;DR: The log-odds for a sample is the sum of the weights of its terminal leafs. The probability of the sample belonging to class 1 is the inverse-logit transformation of the sum. Analogously to ...
user avatar
  • 78.1k
15 votes
Accepted

Why not always use ensemble learning?

In general it is not true that it will always perform better. There are several ensemble methods, each with its own advantages/weaknesses. Which one to use and then depends on the problem at hand. ...
user avatar
  • 13.1k
15 votes

Boosting neural networks

In boosting, weak or or unstable classifiers are used as base learners. This is the case because the aim is to generate decision boundaries that are considerably different. Then, a good base learner ...
user avatar
  • 1,814
14 votes

hard voting versus soft voting in ensemble based methods

Suppose you have probabilities: 0.45 0.45 0.90 Then hard voting would give you a score of 1/3 (1 vote in favour and 2 against), so it would classify as a "negative". Soft voting would give you the ...
user avatar
  • 141
14 votes
Accepted

Is there any theoretical problem with averaging regression coefficients to build a model?

Given that OLS minimizes the MSE of the residuals amongst all unbiased linear estimators (by the Gauss-Markov theorem) , and that a weighted average of unbiased linear estimators (e.g., the estimated ...
user avatar
  • 32.7k
12 votes

US Election results 2016: What went wrong with prediction models?

The USC/LA Times poll has some accurate numbers. They predicted Trump to be in the lead. See The USC/L.A. Times poll saw what other surveys missed: A wave of Trump support http://www.latimes.com/...
user avatar
  • 2,190
12 votes
Accepted

How to stack machine learning models in R

What you're doing here is what I refer to as "Holdout Stacking" (sometimes also called Blending but that term is also used for regular Stacking), where you use a holdout set to generate the training ...
user avatar
11 votes

Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

Ok, after spending some time on googling I found out how I could do the weighting in python even with scikit-learn. Consider the below: I train a set of my regression models (as mentioned SVR, ...
user avatar
11 votes

US Election results 2016: What went wrong with prediction models?

No high ground claimed here. I work in a field (Monitoring and Evaluation) that is as rife with pseudo-science as any other social science you could name. But here's the deal, the polling industry is ...
user avatar
  • 137
11 votes

What if there is no true data-generating process?

Have you heard the "all models are wrong, but some are useful" quote? It's one o the most famous quotes in statistics. Let's use human language as an example. What you say, is a result of ...
user avatar
  • 113k
10 votes
Accepted

When should I not use an ensemble classifier?

The model that is closest to the true data generating process will always be best and will beat most ensemble methods. So if the data come from a linear process lm() will be much superior to random ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible