# Tag Info

### Gradient Boosting Tree vs Random Forest

$\text{error = bias + variance}$ Boosting is based on weak learners (high bias, low variance). In terms of decision trees, weak learners are shallow trees, sometimes even as small as decision stumps (...
• 5,820

### Bagging, boosting and stacking in machine learning

Bagging: parallel ensemble: each model is built independently aim to decrease variance, not bias suitable for high variance low bias models (complex models) an example of a tree based method is ...
• 1,254

### Gradient Boosting Tree vs Random Forest

This question is addressed in this very nice post. Please take a look at it and the references therein. http://fastml.com/what-is-better-gradient-boosted-trees-or-random-forest/ Notice in the article ...
• 13.1k

### Bagging, boosting and stacking in machine learning

Just to elaborate on Yuqian's answer a bit. The idea behind bagging is that when you OVERFIT with a nonparametric regression method (usually regression or classification trees, but can be just about ...
• 2,239
Accepted

### US Election results 2016: What went wrong with prediction models?

In short, polling is not always easy. This election may have been the hardest. Any time we are trying to do statistical inference, a fundamental question is whether our sample is a good ...
• 17.9k

### Won't highly-correlated variables in random forest distort accuracy and feature-selection?

Old thread, but I don't agree with a blanket statement that collinearity is not an issue with random forest models. When the dataset has two (or more) correlated features, then from the point of view ...
• 561
Accepted

### Gradient Boosting for Linear Regression - why does it not work?

What am I missing here? I don't think you're really missing anything! Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as ...
• 33.6k
Accepted

### Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

Actually, scikit-learn does provide such a functionality, though it might be a bit tricky to implement. Here is a complete working example of such an average ...
• 506

### US Election results 2016: What went wrong with prediction models?

There are a number of sources of polling error: You find some people hard to reach This is corrected by doing demographic analysis, then correcting for your sampling bias. If your demographic ...
• 610

### US Election results 2016: What went wrong with prediction models?

This was mentioned in the comments on the accepted answer (hat-tip to Mehrdad), but I think it should be emphasized. 538 actually did this quite well this cycle*. 538 is a polling aggregator that ...
• 411
Accepted

### k-fold Cross validation of ensemble learning

Ensemble learning refers to quite a few different methods. Boosting and bagging are probably the two most common ones. It seems that you are attempting to implement an ensemble learning method called ...
• 636
Accepted

### Won't highly-correlated variables in random forest distort accuracy and feature-selection?

That is correct, but therefore in most of those sub-samplings where variable Y was available it would produce the best possible split. You may try to increase mtry, to make sure this happens more ...
Accepted

### How do ensemble methods outperform all their constituents?

It's not guaranteed. As you say, the ensemble could be worse than the individual models. For example, taking the average of the true model and a bad model would give a fairly bad model. The average ...
• 9,655

### Bagging, boosting and stacking in machine learning

See my ensemble learning blog post Sources for this image: Wikipedia sklearn
• 1,499

### What is the difference between bagging and random forest if only one explanatory variable is used?

The fundamental difference is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, ...
• 319
Accepted

### Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

As noticed in the comments, you’ve re-discovered boosting. Nothing wrong with this approach, but usually it’s easier and safer to use a method already implemented and battle-tested by someone else ...
• 113k
Accepted

### hard voting, soft voting in ensemble based methods

Let's take a simple example to illustrate how both approaches work. Imagine that you have 3 classifiers (1, 2, 3) and two classes (A, B), and after training you are predicting the class of a single ...
• 11.9k

### When should I not use an ensemble classifier?

I do not recommend using an ensemble classifier when your model needs to be interpretable and explainable. Sometimes you need predictions and explanations of the predictions. When you need to ...
• 341

### US Election results 2016: What went wrong with prediction models?

First it was Brexit, now the US election Not really a first, e.g. the French presidential election, 2002 "led to serious discussions about polling techniques". So it's not far-fetched to say these ...
• 42.7k
Accepted

### How does gradient boosting calculate probability estimates?

TL;DR: The log-odds for a sample is the sum of the weights of its terminal leafs. The probability of the sample belonging to class 1 is the inverse-logit transformation of the sum. Analogously to ...
• 78.1k
Accepted

### Why not always use ensemble learning?

In general it is not true that it will always perform better. There are several ensemble methods, each with its own advantages/weaknesses. Which one to use and then depends on the problem at hand. ...
• 13.1k

### Boosting neural networks

In boosting, weak or or unstable classifiers are used as base learners. This is the case because the aim is to generate decision boundaries that are considerably different. Then, a good base learner ...
• 1,814

### hard voting versus soft voting in ensemble based methods

Suppose you have probabilities: 0.45 0.45 0.90 Then hard voting would give you a score of 1/3 (1 vote in favour and 2 against), so it would classify as a "negative". Soft voting would give you the ...
• 141
Accepted

### Is there any theoretical problem with averaging regression coefficients to build a model?

Given that OLS minimizes the MSE of the residuals amongst all unbiased linear estimators (by the Gauss-Markov theorem) , and that a weighted average of unbiased linear estimators (e.g., the estimated ...
• 32.7k

### US Election results 2016: What went wrong with prediction models?

The USC/LA Times poll has some accurate numbers. They predicted Trump to be in the lead. See The USC/L.A. Times poll saw what other surveys missed: A wave of Trump support http://www.latimes.com/...
• 2,190
Accepted

### How to stack machine learning models in R

What you're doing here is what I refer to as "Holdout Stacking" (sometimes also called Blending but that term is also used for regular Stacking), where you use a holdout set to generate the training ...
• 790

### Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

Ok, after spending some time on googling I found out how I could do the weighting in python even with scikit-learn. Consider the below: I train a set of my regression models (as mentioned SVR, ...

### US Election results 2016: What went wrong with prediction models?

No high ground claimed here. I work in a field (Monitoring and Evaluation) that is as rife with pseudo-science as any other social science you could name. But here's the deal, the polling industry is ...
• 137