108
$\begingroup$

First it was Brexit, now the US election. Many model predictions were off by a wide margin, and are there lessons to be learned here? As late as 4 pm PST yesterday, the betting markets were still favoring Hillary 4 to 1.

I take it that the betting markets, with real money on the line, should act as an ensemble of all the available prediction models out there. So it's not far-fetched to say these models didn't do a very good job.

I saw one explanation was voters were unwilling to identify themselves as Trump supporters. How could a model incorporate effects like that?

One macro explanation I read is the rise of populism. The question then is how could a statistical model capture a macro trend like that?

Are these prediction models out there putting too much weight on data from polls and sentiment, not enough from where the country is standing in a 100 year view? I am quoting a friend's comments.

$\endgroup$
  • 9
    $\begingroup$ How to estimate the "unwilling to identify themselves as Trump supporter." effect: Maybe focus groups? This is more of a social science question than statistics per se. $\endgroup$ – kjetil b halvorsen Nov 9 '16 at 18:18
  • 100
    $\begingroup$ Why do the models have to be wrong just because they predicted an outcome that didn't happen? I have a model that says a die is probably not going to show a six, but sometimes it shows a six anyways. $\endgroup$ – dsaxton Nov 9 '16 at 18:20
  • 4
    $\begingroup$ I am not sure if the models really leaned heavily on the wrong side. Were we reading the models' output correctly? I also agree with dsaxton's comment. $\endgroup$ – Richard Hardy Nov 9 '16 at 18:37
  • 7
    $\begingroup$ Some good thoughts at Andrew Gelman's blog here. $\endgroup$ – Richard Hardy Nov 9 '16 at 18:41
  • 22
    $\begingroup$ If the odds were 4:1, the less common outcome should still occur frequently. That is the betting markets could well have been right. $\endgroup$ – gung Nov 9 '16 at 18:43

15 Answers 15

57
$\begingroup$

In short, polling is not always easy. This election may have been the hardest.

Any time we are trying to do statistical inference, a fundamental question is whether our sample is a good representation of the population of interest. A typical assumption that is required for many types of statistical inference is that of having our sample being a completely random sample from the population of interest (and often, we also need samples to be independent). If these assumptions hold true, we typically have good measures of our uncertainty based on statistical theory.

But we definitively do not have these assumptions holding true with polls! We have exactly 0 samples from our population of interest: actual votes cast at election day. In this case, we cannot make any sort of valid inference without further, untestable assumptions about the data. Or at least, untestable until after election day.

Do we completely give up and say "50%-50%!"? Typically, no. We can try to make what we believe are reasonable assumptions about how the votes will be cast. For example, maybe we want to believe that polls are unbiased estimates for the election day votes, plus some certain unbiased temporal noise (i.e., evolving public opinion as time passes). I'm not an expert on polling methods, but I believe this is the type of model 538 uses. And in 2012, it worked pretty well. So those assumptions were probably pretty reasonable. Unfortunately, there's no real way of evaluating those assumptions, outside strictly qualitative reasoning. For more discussion on a similar topic, see the topic of Non-Ignorable missingness.

My theory for why polls did so poorly in 2016: the polls were not unbiased estimates of voter day behavior. That is, I would guess that Trump supporters (and likely Brexit supporters as well) were much more distrustful of pollsters. Remember that Mr. Trump actively denounced polls. As such, I think Trump supporters were less likely to report their voting intentions to pollsters than supporters of his opponents. I would speculate that this caused an unforeseen heavy bias in the polls.

How could analysts have accounted for this when using the poll data? Based on the poll data alone, there is no real way to do this in a quantitative way. The poll data does not tell you anything about those who did not participate. However, one may be able to improve the polls in a qualitative way, by choosing more reasonable (but untestable) assumptions about the relation between polling data and election day behavior. This is non-trivial and the truly difficult part of being a good pollster (note: I am not a pollster). Also note that the results were very surprising to the pundits as well, so it's not like there were obvious signs that the assumptions were wildly off this time.

Polling can be hard.

$\endgroup$
  • 1
    $\begingroup$ @horaceT: how would they know there were biases until they had samples from the population of interest? One of the wrinkles here is that historically, I would guess that this issue is one of noise instead of bias. If both parties have equal levels of non-response, your estimates will be unbiased, just slightly more noisy. But since Mr Trump ran a campaign with heavily negative views of media coverage and polls, much more so than any previous election, the non-response could have easily been very lopsided toward under representing Trump votes. This would be an effect pollsters would have... $\endgroup$ – Cliff AB Nov 10 '16 at 4:08
  • 38
    $\begingroup$ For what it's worth, I still don't think 538 really failed. It gave like a ~30% chance(?) to Trump winning which is pretty damn good -- it means for every 2-3 times it expected to be right, it expected to be wrong 1 time. That's a huge amount of uncertainty, far more than other polls seemed to be willing to admit to. $\endgroup$ – Mehrdad Nov 10 '16 at 7:39
  • 3
    $\begingroup$ This effect is well known: it's called the Bradley effect in the US, and the Shy Tory effect in the UK. $\endgroup$ – Emilio Pisanty Nov 10 '16 at 12:36
  • 15
    $\begingroup$ 538 (and other things like the Sam Wang's PEC) aren't polls. They are models constructed from poll results. All of these models started with basically the same data, but 538 predicted a lot more uncertainty in the results for reasons that Nate Silver discussed pre election extensively. This meant that the 538 chance of a Hillary win was much lower even though it used the same polls. I agree that 538 didn't fail - given its input, a Hillary win with a lot of uncertainty seems like the best prediction even in hindsight. $\endgroup$ – KAI Nov 10 '16 at 17:21
  • 6
    $\begingroup$ I first read the final 538 prediction the morning after the election, and in it Nate Silver quite clearly states that a 3% margin of error would be well in the usual range - and if you look at his chart of a 3% margin of error in Trump's favor, it lines up pretty well with what actually happened. $\endgroup$ – Xiong Chiamiov Nov 11 '16 at 21:07
35
$\begingroup$

There are a number of sources of polling error:

  • You find some people hard to reach

    This is corrected by doing demographic analysis, then correcting for your sampling bias. If your demographic analysis doesn't reflect the things that make people hard to reach, this correction does not repair the damage.

  • People lie

    You can use historical rates at which people lie to pollsters to influence your model. As an example, historically people state they are going to vote 3rd party far more than they actually do on election day. Your corrections can be wrong here.

    These lies can also mess up your other corrections; if they lie about voting in the last election, they may be counted as a likely voter even if they are not, for example.

  • Only the people who vote end up counting

    Someone can have lots of support, but if their supporters don't show up on election day, it doesn't count. This is why we have registered voter, likely voter, etc models. If these models are wrong, things don't work.

  • Polling costs money

    Doing polls is expensive, and if you don't expect (say) Michigan to flip you might not poll it very often. This can lead to surprised where a state you polled 3 weeks before the election looks nothing like that on election day.

  • People change their minds

    Over minutes, hours, days, weeks or months, people change their minds. Polling about "what you would do now" doesn't help much if they change their minds before it counts. There are models that guess roughly the rate at which people change their minds based off historical polls.

  • Herding

    If everyone else states that Hillary is +3 and you get a poll showing Hillary +11 or Donald +1, you might question it. You might do another pass and see if there is an analysis failure. You might even throw it out and do another poll. When you get a Hillary +2 or +4 poll, you might not do it. Massive outliers, even if the statistical model says it happens sometimes, can make you "look bad".

    A particularly crappy form of this happened on election day, where everyone who released a poll magically converged to the same value; they probably where outlier polls, but nobody wants to be the one who said (say) Hillary +11 the day before this election. Being wrong in a herd hurts you less.

  • Expected sampling error

    If you have 1 million people and you ask 100 perfectly random people and half say "Apple" and half say "Orange", the expected error you'd get from sampling is +/- 10 or so, even if none of the above problems occur. This last bit is what polls describe as their margin of error. Polls rarely describe what the above correction factors could introduce as error.


Nate Silver at 538 was one of the few polling aggregators that used conservative (cautious) means to handle the possibility of the above kinds of errors. He factored in the possibility of systemic correlated errors in the polling models.

While other aggregators were predicting a 90%+ chance HC was elected, Nate Silver was stating 70%, because the polls were within "normal polling error" of a Donald victory.

This was a historical measure of model error, as opposed to raw statistical sampling error; what if the model and the corrections to the model were wrong?


People are still crunching the numbers. But, preliminary results indicate a big part of it was turnout models. Donald supporters showed up to the polls in larger numbers, and Hillary supporters in lesser numbers, than the polling models (and exit polls!) indicated.

Latino's voted more for Donald than expected. Blacks voted more for Donald than expected. (Most of both voted for Hillary). White women voted more for Donald than expected (more of them voted for Donald than Hillary, which was not expected).

Voter turnout was low in general. Democrats tend to win when there is high voter turnout, and Republicans when there is low.

$\endgroup$
  • 1
    $\begingroup$ An interesting Turnout problem is that the poll itself influences turnout. Is there a turnout model for that? It should be possible to have a function that takes the survey predicted turnout, and modify it for both sides according to candidate outlook. A candidate far behind may not get extra voters who are more concerned after seeing the poll describe their candidate's prospects as dire, but if your candidate is well ahead, you may not work as hard to get out to vote... It's obviously not a linear function, but it should be measurable. $\endgroup$ – BenPen Nov 10 '16 at 21:11
  • 2
    $\begingroup$ +1 from me just for mentioning herding and explaining it well. As I went over in my answer, I was very suspicious herding might be happening starting around the 5th or so (3ish days from the election) based on the 538 graph. I'm guessing we'll find out more about what the errors really were in the upcoming days. (You know you're a nerd when you're obsessively refreshing a web page to contemplate the second derivative of a graph curve there). $\endgroup$ – T.E.D. Nov 10 '16 at 21:51
  • $\begingroup$ I don't know how you account for it, but I think there's a stigma associated with Trump that would make it hard to properly quantify his actual support and would only show up in the actual election results. I like to think of it as the bumper sticker corollary: George W. Bush and Obama were both 2 term presidents, but while an Obama bumper sticker is widespread and adorned on cars with pride, a Bush bumper sticker was like a 4 leaf clover. There's certain candidates where open support draws too much heat and vitriol from the opposition and the support is very low-key. $\endgroup$ – coburne Nov 11 '16 at 18:09
  • 3
    $\begingroup$ @coburne There was no evidence of that in the primaries; Trump supporters where not shy about it. Bush bumper stickers where popular in different areas than Obama bumper stickers. $\endgroup$ – Yakk Nov 11 '16 at 18:22
  • $\begingroup$ @coburne - What you are talking about is called The Bradley Effect. There is a huge debate over whether it even exists. There was one study though that supposedly found its power roughly proportional to how much racially-charged rhetoric was used in the campaign. I don't think there's much debate that a lot of that was used in this one. $\endgroup$ – T.E.D. Nov 16 '16 at 15:46
31
$\begingroup$

This was mentioned in the comments on the accepted answer (hat-tip to Mehrdad), but I think it should be emphasized. 538 actually did this quite well this cycle*.

538 is a polling aggregator that runs models against each state to try to predict the winner. Their final run gave Trump about a 30% chance of winning. That means if you ran three elections with data like this, you'd expect Team Red to win one of them. That isn't really that small of a chance. Its certainly a big enough one that I took precautions (eg: The Friday before I asked for Wednesday the 9th off at work, considering the likelihood of it being close enough to be a late night).

One thing 538 will tell you if you hang out there is that if polls are off, there's a good chance they will all be off in the same direction. This is for a couple of reasons.

  • Likely voter models. Polls have to adjust for the the types of voters who will actually show up on election day. We have historical models, but this was obviously not your typical pair of candidates, so predicting based on past data was always going to be a bit of a crapshoot.
  • Late election herding. Nobody wants to be the poll that blew the election the worst. So while they don't mind being an outlier in the middle of a campaign, at the end all the polls tend to tweak themselves so that they say the same thing. This is one of the things that was blamed for the polls being so egregiously off in Eric Cantor's surprise loss in 2014, and for the surprisingly close results of the 2014 Virginia Senate race as well.

* - 538 has now posted their own analysis. It mostly jibes with what is said above, but is worth reading if you want a lot more details.


Now a bit of personal speculation. I was actually skeptical of 538's final % chances for its last 3 days. The reason goes back to that second bullet above. Let's take a look at the history of their model for this election (from their website)

enter image description here

(Sadly, the labels obscure it, but after this the curves diverged again for the last three days, out to more than a 70% chance for Clinton)

The pattern we see here is repeated divergence followed by decay back toward a Trump lead. The Clinton bubbles were all caused by events. The first was the conventions (normally there's a couple of days lag after an event for it to start showing up in the polling). The second seems to have been kicked off by the first debate, likely helped along by the TMZ tape. Then there's the third inflection point I've marked in the picture.

It happened on November 5, 3 days before the election. What event caused this? A couple days before that was another email-flareup, but that shouldn't have worked in Clinton's favor.

The best explanation I could come up with at the time was poll herding. It was only 3 days until the election, 2 days until the final polls, and pollsters would be starting to worry about their final results. The "conventional wisdom" this entire election (as evidenced by the betting models) was an easy Clinton win. So it seemed a distinct possibility that this wasn't a true inflection at all. If that were the case, the true curve from Nov 5 on was quite likely a continuation of this one towards convergence.

It would take a better mathematician than I to estimate the curve forward here without this suspicious final inflection point, but eyeballing it I think Nov 8 would have been near the crossover point. In front or behind depends on how much of that curve was actually real.

Now I can't say for sure this is what happened. There are other very plausible explanations (eg: Trump got his voters out far better than any pollster expected) But it was my theory for what was going on at the time, and it certainly proved predictive.

$\endgroup$
  • 1
    $\begingroup$ I think this weird poll inflection in the last few days would have been better analyzed, but Clinton supporters saw what they wanted to see, and Trump supporters had long since quit heeding the polls. Hopefully somebody will do it now. $\endgroup$ – T.E.D. Nov 10 '16 at 16:22
  • $\begingroup$ I thought the last days normalised slightly due to the Comey’s statement that the new emails did not constitute cause for renewed criminal investigation. $\endgroup$ – Konrad Rudolph Nov 10 '16 at 19:46
  • $\begingroup$ @KonradRudolph - That was the explanation that I heard given for that inflection at the time. The problem is the statement in question did not come out until Nov 6, and the suspicious polling inflection point occurred a day earlier (see the marker in the picture above). Also, the timing is wrong for the drop being entirely explained by Comey, so there's no logical reason his "nevermind" statement would have stopped it (much less turned it around). $\endgroup$ – T.E.D. Nov 10 '16 at 19:58
  • 2
    $\begingroup$ The problem with 538 is not so much their model as the quality of the polling data that went into it. The data make clear that this was not a case of sampling error (which is quite small when you average polls that each have decent sample sizes). washparkprophet.blogspot.com/2016/11/what-polls-got-wrong.html Instead, the problem is either biased sampling in the lion's share of the polls, or systemic untruthfulness from poll respondents (due to social disapproval of Trump) or both. But, 538 gets kudos for recognizing in their model that polls in different states aren't independent. $\endgroup$ – ohwilleke Nov 11 '16 at 1:07
  • $\begingroup$ @ohwilleke - Right. As one of the other answers said, GIGO. That's what I figured was likely happening with that weird unexplained inflection point. The question is the source of the "garbage" in the input polls. $\endgroup$ – T.E.D. Nov 11 '16 at 1:16
17
$\begingroup$

First it was Brexit, now the US election

Not really a first, e.g. the French presidential election, 2002 "led to serious discussions about polling techniques".

So it's not far-fetched to say these models didn't do a very good job.

Garbage in, garbage out.

I saw one explanation was voters were unwilling to identify themselves as Trump supporter. How could a model incorporate effects like that?

See response bias, and in particular social desirability bias. Other interesting reads: silent majority and Bradley effect.

$\endgroup$
  • 2
    $\begingroup$ Sure, garbage in garbage out. But how does one recognize the predictors were garbage, and do "variable selection" to get ride of them? $\endgroup$ – horaceT Nov 9 '16 at 18:29
  • 6
    $\begingroup$ @horaceT as you can see, this is very hard and sometimes could be impossible. FiveThirtyEight.com had very decent methodology and high-quality model, using diverse data and correcting for multiple bias. The day before election it gave 71.4% probability that Hilary Clinton will win... $\endgroup$ – Tim Nov 9 '16 at 18:36
  • 1
    $\begingroup$ @horaceT I would focus on data collection, since that seems to be the issue. The social desirability bias page contain some ideas to improve it. $\endgroup$ – Franck Dernoncourt Nov 9 '16 at 18:40
  • 1
    $\begingroup$ @horaceT moreover, if almost every pool said that Clinton leads only a madman would argue that they all are wrong... It would be very hard to justify such model. $\endgroup$ – Tim Nov 9 '16 at 18:46
  • 1
    $\begingroup$ I would be curious to know how accurate the polls' predictions were for voter turnout (e.g. based on demographics). I could imagine that if many polls predicting a "significant lead", turnout could be suppressed (e.g. similar to an observer effect)? $\endgroup$ – GeoMatt22 Nov 11 '16 at 5:11
12
$\begingroup$

The USC/LA Times poll has some accurate numbers. They predicted Trump to be in the lead. See The USC/L.A. Times poll saw what other surveys missed: A wave of Trump support

http://www.latimes.com/politics/la-na-pol-usc-latimes-poll-20161108-story.html

enter image description here

They had accurate numbers for 2012 as well.

You may want to review: http://graphics.latimes.com/usc-presidential-poll-dashboard/

And NY Times complained about their weighting: http://www.nytimes.com/2016/10/13/upshot/how-one-19-year-old-illinois-man-is-distorting-national-polling-averages.html

LA Times' response: http://www.latimes.com/politics/la-na-pol-daybreak-poll-questions-20161013-snap-story.html

$\endgroup$
  • 26
    $\begingroup$ This poll had Trump winning the popular vote by 3.2%, but Clinton seems to have won by .1%. So I don't see how you can say they had accurate numbers. $\endgroup$ – Winston Ewert Nov 10 '16 at 1:35
  • 3
    $\begingroup$ Just a slight note - would you really expect any statistic to be within less than 3.2% of an error window? $\endgroup$ – AnoE Nov 10 '16 at 13:49
  • 9
    $\begingroup$ Problems with this poll as an example are 1) Its polling the wrong thing. Popular vote is correlated with winning the Presidency, but that's not how its decided. 2) It got the topline wrong. Clinton won what it is measuring, not Trump. 3) It was off by the same 3ish points most of the other polls were, just in a different direction. $\endgroup$ – T.E.D. Nov 10 '16 at 14:56
  • 5
    $\begingroup$ ...actually, it looks like Clinton may finish about a full point ahead of Trump in the popular vote, which means this poll was off by 4, not 3. So in theory a similar poll that had her winning by 3 points would have been twice as accurate as this one (off by only 2 points rather than 4). $\endgroup$ – T.E.D. Nov 10 '16 at 20:15
  • 8
    $\begingroup$ The LA Times poll was correct by accident: the over-weighted 19-year-old counterbalanced the under-weighted white rural vote. $\endgroup$ – Mark Nov 10 '16 at 21:07
11
$\begingroup$

No high ground claimed here. I work in a field (Monitoring and Evaluation) that is as rife with pseudo-science as any other social science you could name.

But here's the deal, the polling industry is supposedly in 'crisis' today because it got the US election predictions so wrong, social science in general has a replicability 'crisis' and back in the late 2000's we had a world financial 'crisis' because some practitioners believed that sub-prime mortgage derivatives were a valid form of financial data (if we give them the benefit of the doubt...).

And we all just blunder on regardless. Everyday I see the most questionable of researcher constructs used as data collection approaches, and therefore eventually used as data (everything from quasi-ordinal scales to utterly leading fixed response categories). Very few researchers even seem to realize they need to have a conceptual framework for such constructs before they can hope to understand their results. It is as if we have looked at market 'research' approaches and decided to adopt only the worst of their mistakes, with the addition of a little numerology on the side.

We want to be considered 'scientists', but the rigor is all a bit too hard to be bothered with, so we collect rubbish data and pray to the Loki-like god of statistics to magically over-ride the GIGO axiom.

But as the heavily quoted Mr Feynman points out:

“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong”.

There are better ways to handle the qualitative data which we are often stuck with, but they take a bit more work and those nice researcher constructs are often way easier to feed into SPSS. Convenience seems to trump science every time (no pun intended).

In short, if we do not start to get serious about raw data quality, I think we are just wasting everyone's time and money, including our own. So does anyone want to collaborate on a 'data quality initiative' in relation to social science methods (yes, there is plenty in the text books about such things, but no one seems to pay attention to that source after their exams).

Whoever has the most academic gravitas gets to be the lead! (It won't be me.)

Just to be clear about my answer here: I see serious fundamental issues with 'contrived' raw data types so often, that I would like to suggest a need to start at the beginning. So even before we worry about sampling or which tests to run on the data, we need to look at the validity/limitations of the data types we collect in relation to the models we are proposing. Otherwise the overall predictive model is incompletely defined.

$\endgroup$
  • 2
    $\begingroup$ Taken far afield I'm sure, can you give examples of the questionable researcher constructs. $\endgroup$ – horaceT Nov 10 '16 at 1:23
  • 4
    $\begingroup$ I don't necessarily disagree with a lot of your points. But I just want to point out that in the case of polling, I think every pollster is extremely aware of the limitations due to data quality, but don't really have any options to improve it (see my answer). Your answer seems to suggests that pollsters want push out any answer, not caring at all about data quality. I think pollster care a lot about data quality, but also realize that the best they can get has serious potential flaws. Do you give up ("50%-50%!") or try to build something that might be reasonable? $\endgroup$ – Cliff AB Nov 10 '16 at 4:31
  • $\begingroup$ my response to comments was necessarily a bit long, so added it as a new answer $\endgroup$ – colin Nov 10 '16 at 23:38
9
$\begingroup$

Polls tend to have an error margin of 5% that you can't really get rid of, because it's not a random error, but a bias. Even if you average across many polls, it does not get much better. This has to do with misrepresented voter groups, lack of mobilization, inability to go to the vote on a workday, unwillingness to answer, unwillingness to answer right, spontaneous last-minute decisions, ... because this bias tends to be "correlated" across polls, you can't get rid of it with more polls; you also can't get rid of it with larger sample sizes; and you don't appear to be able to predict this bias either, because it changes too fast (and we elect presidents too rarely).

Due to the stupid winner-takes-all principle still present in almost all states, an error of 5% can cause very different results: Assume the polls always predicted 49-51, but the real result was 51-49 (so an error of just 2%), the outcome is 100% off; because of winner-takes-it-all.

If you look at individual states, most results are within the predicted error margins!

Probably the best you can do is sample this bias (+-5%), apply the winner-takes-all extremes, then aggregate the outcomes. This is probably similar to what 538 did; and in 30% of the samples Donald Trump won...

$\endgroup$
  • 9
    $\begingroup$ I call this the "lunatic fringe principle" of polling: in any survey question, 5% of all respondents will give a crazy answer. Like any empirical principle it has exceptions, but it has stood up well for decades in helping make sense of poll results. $\endgroup$ – whuber Nov 10 '16 at 21:22
  • 1
    $\begingroup$ If it only were just a "crazy" answer. The problem is that it is systematic not "random crazy". You could consider the election a binary poll, and what "crazy answers" could you expect in binary? But apparently, a lot of people deliberately (?) give a wrong answer, or decide differently when actually in the booth, or then don't go to the elections, ... $\endgroup$ – Anony-Mousse Nov 10 '16 at 21:36
  • 3
    $\begingroup$ @Anony-Mousse no matter how accurate it may or may not be, I fail to see how juvenile name-calling is relevant to statistical analysis. $\endgroup$ – Jared Smith Nov 11 '16 at 2:44
  • $\begingroup$ Oh, it's a priceless story. On some days, you have to laugh, rather than worry why prediction results are inaccurate. $\endgroup$ – Anony-Mousse Nov 11 '16 at 6:59
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – gung May 27 at 18:38
7
$\begingroup$

The reliance on data analysis had a huge impact in strategic campaign decisions, journalistic coverage, and ultimately in individual choices. What could possibly go wrong when the Clinton campaign's decisions were informed by no other than $\small 400,000$ daily simulations on the secret Ada algorithm?

In the end, it exposed a colossal failure of numerical analysis to make up for lack of knowledge of the subject matter. People were ashamed of themselves to explicitly embrace the winning candidate for obvious reasons.

The worst computer model could have gotten closer to the outcome if anybody had bothered to conduct a preliminary poll face to face, knocking on doors. Here is an example: the Trafalgar Group (no affiliation or knowledge other than what follows) had Trump leading in PA, FL, MI, GA, UT and NV (this latter state went ultimately blue) one day prior to the election. What was the magic?

a combination of survey respondents to both a standard ballot test and a ballot test guaging [sic] where respondent's neighbors stand. This addresses the underlying bias of traditional polling, wherein respondents are not wholly truthful about their position regarding highly controversial candidates.

Pretty low-tech, including the lack of spell-check, showing in numbers a lot about human nature. Here is the discrepancy in PA:

enter image description here

Historic Pennsylvania - so far from being perceived as the final straw in the Democratic defeat just hours prior to this closing realization at 1:40 am on November 9, 2016:

enter image description here

$\endgroup$
  • 2
    $\begingroup$ Asking about the neighbours' voting intention is brilliant - it seems to me one of those clever tricks sometimes used in Statistics, which allow correcting (to a degree , at least) for a seemingly hopeless bias. Thanks for writing about that, very interesting! $\endgroup$ – DeltaIV Nov 11 '16 at 22:18
5
$\begingroup$

One of the reasons for poll inaccurracy in the US election, besides some people for whatever reason don´t say the truth is, that the "winner takes it all" effect makes predictions even less easier. A 1% difference in one state can lead to a complete shift of a state and influence the whole outcome very heavily. Hillary had more voters just like Al Gore vs Bush.

The Brexit referendum was not a normal election and therefore also harder to predict (No good historical data and everyone was like a first time voter on this matter). People who for decades vote for the same party stabilize predictions.

$\endgroup$
  • 2
    $\begingroup$ Very good observation. There were clear states for each side and swing states. While their number was low, the effect on a small change there is big in number of votes. It's a very convoluted, historically grown voting scheme in the US. $\endgroup$ – Trilarion Nov 14 '16 at 9:41
4
$\begingroup$

(Just answering this bit, as the other answers seem to have covered everything else.)

As late as 4 pm PST yesterday, the betting markets were still favoring Hillary 4 to 1. I take it that the betting markets, with real money on the line, should act as an ensemble of all the available prediction models out there.

No... but indirectly yes.

The betting markets are designed so the bookies make a profit whatever happens. E.g. say the current odds quoted were 1-4 on Hilary, and 3-1 on Trump. If the next ten people all bet \$10 on Hilary, then that \$100 taken in is going to cost them \$25 if Hilary wins. So they shorten Hilary to 1-5, and raise Trump to 4-1. More people now bet on Trump, and balance is restored. I.e. it is purely based on how people bet, not on the pundits or the prediction models.

But, of course, the customers of the bookies are looking at those polls, and listening to those pundits. They hear that Hilary is 3% ahead, a dead cert to win, and decide a quick way to make \$10 is to bet \$40 on her.

Indirectly the pundits and polls are moving the odds.

(Some people also notice all their friends at work are going to vote Trump, so make a bet on him; others notice all their Facebook friend's posts are pro-Hilary, so make a bet on her, so there is a bit of reality influencing them, in that way.)

$\endgroup$
2
$\begingroup$

It is not surprising that these efforts failed, when you consider the disparity between what information the models have access to and what information drives behavior at the polling booth. I'm speculating, but the models probably take into account:

  • a variety of pre-election polling results
  • historical state leanings (blue/red)
  • historical results of prior elections with current state leanings/projections

But, pre-election polls are unreliable (we've seen constant failures in the past), states can flip, and there haven't been enough election cycles in our history to account for the multitude of situations that can, and do, arise.

Another complication is the confluence of the popular vote with the electoral college. As we saw in this election, the popular vote can be extremely close within a state, but once the state is won, all votes go to one candidate, which is why the map has so much red.

$\endgroup$
1
$\begingroup$

The polling models didn't consider how many Libertarians might switch from Johnson to Trump when it came to actual voting. The states which were won by a thin margin were won based on which percentage of the vote Johnson got. PA (which pushed Trump past 270 on the election night) gave only 2% to Johnson. NH (which went to Clinton) gave 4%+ to Johnson. Johnson was polling at 4%-5% the day before the election and he got roughly 3% on the day of the election.

So why did Libertarians, all of a sudden, switch on the day of the election? No one considered what was the central issue to Libertarian voters. They tend to view literal interpretation of the Constitution as canon. Most people who voted for Clinton did not think that her dismissiveness of the law was a high enough priority to consider. Certainly, not higher than everything which they didn't like about Trump.

Regardless of whether her legal troubles were important or not to others, they would be important to Libertarians. They would put a very high priority on keeping out of office someone who viewed legal compliance as optional, at best. So, for a large number of them, keeping Clinton out of office would become a higher priority than making a statement that Libertarian philosophy is a viable political philosophy.

Many of them may not have even liked Trump, but if they thought that he would be more respectful of the rule of law than Clinton would be, pragmatism would have won over principles for a lot of them and caused them to switch their vote when it came time to actually vote.

$\endgroup$
  • $\begingroup$ NH has all the people from the free state project living there. Thriving libertarian party and active supporters. $\endgroup$ – John Nov 11 '16 at 21:58
  • $\begingroup$ @John, NH Libertarians stuck with Johnson (4%). Trump lost the state by 1%. $\endgroup$ – Dmitry Rubanovich Nov 11 '16 at 22:01
  • $\begingroup$ I understand that. I was trying to explain that the libertarian party is strong in NH. $\endgroup$ – John Nov 11 '16 at 22:27
  • $\begingroup$ @John, but it's not just NH. Minnesota: Johnson 4%, Trump lost by 2%; NV (a harder argument to make, but still the trend holds): Johnson 3.5%, Trump lost by 2%; Maine: Johnson 5%, Trump lost by 3%; Colorado Johnson 5%, Trump lost by 3%. $\endgroup$ – Dmitry Rubanovich Nov 11 '16 at 22:44
  • $\begingroup$ Afaik, polls ask about possible vote-switching and forecasts take it into consideration. Do you have any information that suggests that before the election there was any such information that was not taken into consideration by any forecast or is this a pure speculation? $\endgroup$ – Tim Nov 16 '16 at 6:24
1
$\begingroup$

Polls are not historical trends. A Bayesian would inquire as to the historical trends. Since Abraham Lincoln, there has been a Republican party and a Democratic party holding the presidential office. The trend for party change 16 times since then from Wikipedia has the following cumulative mass function

enter image description here

where time in years to a change of presidential party is on the $x$-axis. After 8-years of a party in power, the odds are 68.75% that the voters vote for a change, just over 2 to 1. Moreover, since the 1860 election, Republicans have held the presidency 59% of the time versus 41% for Democrats.

What made journalists, the Democratic party, and the pollsters think that the odds were in favor of liberals winning was perhaps wishful thinking. Behavior may be predictable, within limits, but in this case the Democrats were wishing that people would not vote for a change, and from a historical perspective, it seems more likely there would be one than not.

$\endgroup$
0
$\begingroup$

I think poll results were extrapolated to the extent of the public assuming the voter demographics will be similar to poll taker demographics and would be a good representation of the whole population. For example, if 7 out of 10 minorities supported Hillary in the polls, and if that minority represents 30% of the US population, the majority of polls assumed 30% of voters will be represented by that minority and translated to that 21% gain for Hillary. In reality, white, middle-to-upper class males were better represented among the voters. Less than 50% of eligible people voted and this didn't translate into 50% off all genders, races, etc.

Or, polls assumed perfect randomization and based their models on that but in reality the voter data was biased toward older middle-to-upper class males.

Or, the polls didn't exactly assume perfect randomization but their extrapolation parameters underestimated the heterogeneity of voter demographics.

ETA: Polls of previous two elections performed better because of increased attention to voting by groups that aren't usually represented well.

$\endgroup$
  • $\begingroup$ As far as I know, all polls base their predictions on 'likely voters'. I can't imagine polls that assume a 20 year old has the same chance to vote as a 70 year old. More central seems the problem: how likely is someone to vote? $\endgroup$ – dimpol Nov 10 '16 at 12:22
  • $\begingroup$ Accounting for the demographics is the easiest part. You just reweight your sample population to the actual population. Accounting for voter turn-out and the biases mentioned in the other answers is a lot harder, though. $\endgroup$ – Graipher Nov 10 '16 at 17:21
  • $\begingroup$ There is a fair amount of variety in how pollsters address these issues. Some demographically rebalance or rebalance based on party affiliation, others don't. But, since there is variation on the models use polling averages, the end result should be robust to problems particular to one method of doing this that is not shared by other polls, particularly after controlling for historical partisan biases (i.e. house effects) of particular polling operations. The problems in the average polling results have to come from shared methods or effects, not methods particular to each poll. $\endgroup$ – ohwilleke Nov 11 '16 at 1:21
0
$\begingroup$

HoraceT and CliffAB (sorry too long for comments) I’m afraid I have a lifetime of examples, which have also taught me that I need to be very careful with their explanation, if I wish to avoid offending people. So while I don’t want your indulgence, I do ask for your patience. Here goes:

To start with an extreme example, I once saw a proposed survey question that asked illiterate village farmers (South East Asia), to estimate their ‘economic rate of return’. Leaving the response options aside for now, we can hopefully all see that this a stupid thing to do, but consistently explaining why it is stupid is not so easy. Yes, we can simply say that it is stupid because the respondent will not understand the question and just dismiss it as a semantic issue. But this is really not good enough in a research context. The fact that this question was ever suggested implies that researchers have inherent variability on what they consider ‘stupid’. To address this more objectively, we must step back and transparently declare a relevant framework for decision making about such things. There are many such options, and I will use one that I sometimes find useful - but have no intent of defending here (I actively encourage anyone to think of others, as it means you are already starting down the road to better conceptualizations).

So, let’s transparently assume that we have two basic information types we can use in analyses: qualitative and quantitative. And that the two are related by a transformative process, such that all quantitative information started out as qualitative information but went through the following (oversimplified) steps:

  1. Convention setting (e.g. we all decided that [regardless of how we individually perceive it], that we will all call the colour of a daytime open sky “blue”.)
  2. Classification (e.g. we assess everything in a room by this convention and separate all items into ‘blue’ or ‘not blue’ categories)
  3. Count (we count/detect the ‘quantity’ of blue things in the room)

Note that (under this model) without step 1, there is no such thing as a quality and if you don’t start with step 1, you can never generate a meaningful quantity.

Once stated, this all looks very obvious, but it is such sets of first principles that (I find) are most commonly overlooked and therefore result in ‘Garbage-In’.

So the ‘stupidity’ in the example above becomes very clearly definable as a failure to set a common convention between the researcher and the respondents. Of course this is an extreme example, but much more subtle mistakes can be equally garbage generating. Another example I have seen is a survey of farmers in rural Somalia, that asked “How has climate change affected your livelihood?” Again putting response options aside for the moment, I would suggest that even asking this of farmers in the Mid-West of the United States would constitute a serious failure to use a common convention between researcher and respondent (i.e. as to what is being measured as ‘climate change’).

Now let’s move on to response options. By allowing respondents to self-code responses from a set of multiple choice options or similar construct you are pushing this ‘convention’ issue into this aspect of questioning as well. This may be fine if we all stick to effectively ‘universal’ conventions in response categories (e.g. question: what town do you live in? response categories: list of all towns in research area [plus ‘not in this area’]). However, many researchers actually seem to take pride in the subtle nuancing of their questions and response categories to meet their needs. In the same survey that the ‘rate of economic return’ question appeared, the researcher also asked the respondents (poor villagers), to provide which economic sector they contributed to: with response categories of ‘production’, ‘service’, ‘manufacturing’ and ‘marketing’. Again a qualitative convention issue obviously arises here. However, because he made the responses mutually exclusive, such that respondents could only choose one option (because “it is easier to feed into SPSS that way”), and village farmers routinely produce crops, sell their labour, manufacture handicrafts and take everything to local markets themselves, this particular researcher did not just have a convention issue with his respondents, he had one with reality itself.

This is why old bores like myself will always recommend the more work intensive approach of applying coding to data post-collection - as at least you can adequately train coders in researcher-held conventions (and note that trying to impart such conventions to respondents in ‘survey instructions’ is a mug’s game –just trust me on this one for now). Also note also that if you accept the above ‘information model’ (which, again, I am not claiming you have to), it also shows why quasi-ordinal response scales have a bad reputation. It is not just the basic maths issues under the Steven’s convention (i.e. you need to define a meaningful origin even for ordinals, you can’t add and average them, etc. etc.), it is also that they have often never been through any transparently declared and logically consistent transformative process that would amount to ‘quantification’ (i.e. an extended version of the model used above that also encompasses generation of ‘ordinal quantities’ [-this is not hard to do]). Anyway, if it does not satisfy the requirements of being either qualitative or quantitative information, then the researcher is actually claiming to have discovered a new type of information outside the framework, and the onus is therefore on them to explain its fundamental conceptual basis fully (i.e. transparently define a new framework).

Finally let’s look at sampling issues (and I think this aligns with some of the other answers already here). For example, if a researcher wants to apply a convention of what constitutes a ‘liberal’ voter, they need to be sure that the demographic information they use to choose their sampling regime is consistent with this convention. This level is usually the easiest to identify and deal with as it is largely within researcher control and is most often the type of assumed qualitative convention that is transparently declared in research. This is also why it is the level usually discussed or critiqued, while the more fundamental issues go unaddressed.

So while pollers stick to questions like ‘who do you plan to vote for at this point in time?’, we are probably still ok, but many of them want to get much ‘fancier’ than this…

$\endgroup$

protected by Tim Nov 10 '16 at 18:19

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.