Skip to main content
56 votes
Accepted

US Election results 2016: What went wrong with prediction models?

In short, polling is not always easy. This election may have been the hardest. Any time we are trying to do statistical inference, a fundamental question is whether our sample is a good ...
Cliff AB's user avatar
  • 21.4k
55 votes

Won't highly-correlated variables in random forest distort accuracy and feature-selection?

Old thread, but I don't agree with a blanket statement that collinearity is not an issue with random forest models. When the dataset has two (or more) correlated features, then from the point of view ...
GDB's user avatar
  • 651
39 votes
Accepted

Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

Actually, scikit-learn does provide such a functionality, though it might be a bit tricky to implement. Here is a complete working example of such an average ...
constt's user avatar
  • 506
36 votes

US Election results 2016: What went wrong with prediction models?

There are a number of sources of polling error: You find some people hard to reach This is corrected by doing demographic analysis, then correcting for your sampling bias. If your demographic ...
Yakk's user avatar
  • 920
31 votes

US Election results 2016: What went wrong with prediction models?

This was mentioned in the comments on the accepted answer (hat-tip to Mehrdad), but I think it should be emphasized. 538 actually did this quite well this cycle*. 538 is a polling aggregator that ...
T.E.D.'s user avatar
  • 411
29 votes
Accepted

How do ensemble methods outperform all their constituents?

It's not guaranteed. As you say, the ensemble could be worse than the individual models. For example, taking the average of the true model and a bad model would give a fairly bad model. The average ...
Flounderer's user avatar
  • 10.6k
28 votes

What is the difference between bagging and random forest if only one explanatory variable is used?

The fundamental difference is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, ...
user3303020's user avatar
21 votes
Accepted

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

As noticed in the comments, you’ve re-discovered boosting. Nothing wrong with this approach, but usually it’s easier and safer to use a method already implemented and battle-tested by someone else ...
Tim's user avatar
  • 140k
19 votes
Accepted

Hard voting, soft voting in ensemble based methods

Let's take a simple example to illustrate how both approaches work. Imagine that you have 3 classifiers (1, 2, 3) and two classes (A, B), and after training you are predicting the class of a single ...
mkt's user avatar
  • 18.9k
19 votes
Accepted

How does gradient boosting calculate probability estimates?

TL;DR: The log-odds for a sample is the sum of the weights of its terminal leafs. The probability of the sample belonging to class 1 is the inverse-logit transformation of the sum. Analogously to ...
Sycorax's user avatar
  • 92.3k
17 votes

US Election results 2016: What went wrong with prediction models?

First it was Brexit, now the US election Not really a first, e.g. the French presidential election, 2002 "led to serious discussions about polling techniques". So it's not far-fetched to say these ...
Franck Dernoncourt's user avatar
17 votes
Accepted

Is there any theoretical problem with averaging regression coefficients to build a model?

Given that OLS minimizes the MSE of the residuals amongst all unbiased linear estimators (by the Gauss-Markov theorem) , and that a weighted average of unbiased linear estimators (e.g., the estimated ...
jbowman's user avatar
  • 40.1k
15 votes

hard voting versus soft voting in ensemble based methods

Suppose you have probabilities: 0.45 0.45 0.90 Then hard voting would give you a score of 1/3 (1 vote in favour and 2 against), so it would classify as a "negative". Soft voting would give you the ...
darkwatch's user avatar
  • 251
14 votes

What is the difference between bagging and random forest if only one explanatory variable is used?

I would like to provide clarification, there is a disctinction between bagging and bagged trees. Bagging (bootstrap + aggregating) is using an ensemble of models where: each model uses a ...
Fazzolini's user avatar
  • 265
14 votes
Accepted

How to stack machine learning models in R

What you're doing here is what I refer to as "Holdout Stacking" (sometimes also called Blending but that term is also used for regular Stacking), where you use a holdout set to generate the training ...
Erin LeDell's user avatar
13 votes

What if there is no true data-generating process?

Have you heard the "all models are wrong, but some are useful" quote? It's one o the most famous quotes in statistics. Let's use human language as an example. What you say, is a result of ...
Tim's user avatar
  • 140k
12 votes

US Election results 2016: What went wrong with prediction models?

The USC/LA Times poll has some accurate numbers. They predicted Trump to be in the lead. See The USC/L.A. Times poll saw what other surveys missed: A wave of Trump support http://www.latimes.com/...
Jon's user avatar
  • 2,350
11 votes
Accepted

State-of-the-art ensemble learning algorithm in pattern recognition tasks?

State-of-the-art algorithms may differ from what is used in production in the industry. Also, the latter can invest in fine-tuning more basic (and often more interpretable) approaches to make them ...
Franck Dernoncourt's user avatar
11 votes

US Election results 2016: What went wrong with prediction models?

No high ground claimed here. I work in a field (Monitoring and Evaluation) that is as rife with pseudo-science as any other social science you could name. But here's the deal, the polling industry is ...
colin's user avatar
  • 137
10 votes

Gradient Boosting for Linear Regression - why does it not work?

The least squares projection matrix is given by $X(X^{T}X)^{-1}X^{T}$ We can use this to directly obtain our predicted values $\hat{y}$, e.g. $\hat{y} = X(X^{T}X)^{-1}X^{T}y $ Let's say you fit a ...
kirtap's user avatar
  • 431
10 votes
Accepted

Bootstrapping dataset with imbalanced classes

One method you can try is a form of "stratified"-bootstrap. You can subsample from each group separately, even un-proportionally. Doing so will result in estimation of the empirical distribution of ...
tmrlvi's user avatar
  • 1,057
10 votes

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

As was mentioned in the comments this idea of iteratively learning from previous model errors is at the core of boosting methodologies like Adaboost or gradient boosting. As you theorize the idea is ...
Tylerr's user avatar
  • 1,562
10 votes

What if there is no true data-generating process?

Looking at it the other way, if there were no true data generating process, how did the data get generated? The inability of standard estimating techniques to accurately approximate the true data-...
Dikran Marsupial's user avatar
10 votes

Ensemble classifiers trained using different sets of features

You can certainly stack your classifiers: apply your classifiers to yield classification probabilities in-sample, then train another model on the ground truth, using the two probabilities as ...
Stephan Kolassa's user avatar
9 votes

US Election results 2016: What went wrong with prediction models?

Polls tend to have an error margin of 5% that you can't really get rid of, because it's not a random error, but a bias. Even if you average across many polls, it does not get much better. This has to ...
Has QUIT--Anony-Mousse's user avatar
9 votes

What is the difference between bagging and random forest if only one explanatory variable is used?

Bagging in general is an acronym like work that is a portmanteau of Bootstrap and aggregation. In general if you take a bunch of bootstrapped samples of your original dataset, fit models $M_1, M_2, \...
Lucas Roberts's user avatar
8 votes
Accepted

Can I use output of classifier A as feature for classifier B?

Yes, using outputs of one model as inputs of another is possible, and this concept is used to some extent in some approaches. If you stack models on top of each other, in the application case the ...
geekoverdose's user avatar
  • 3,901
8 votes

Stacking without splitting data

You have to have real, out-of-sample, predictions as the input to your blender, otherwise your blender is not learning about, and thereby improving, prediction accuracy - but instead learning about, ...
jbowman's user avatar
  • 40.1k
8 votes
Accepted

Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

The main difference between AdaBoost and other "generic" boosting algorithms is that AdaBoost uses the (deviance) residuals as weights while "generic" gradient boosting algorithms use the residuals as ...
usεr11852's user avatar
  • 44.8k
7 votes
Accepted

Ensemble Learning: Why is Model Stacking Effective?

Think of ensembling as basically an exploitation of the central limit theorem. The central limit theorem loosely says that, as the sample size increases, the mean of the sample will become an ...
Doug Dame's user avatar
  • 156

Only top scored, non community-wiki answers of a minimum length are eligible