US Election results 2016: What went wrong with prediction models?

Question

First it was Brexit, now the US election. Many model predictions were off by a wide margin, and are there lessons to be learned here? As late as 4 pm PST yesterday, the betting markets were still favoring Hillary 4 to 1.

I take it that the betting markets, with real money on the line, should act as an ensemble of all the available prediction models out there. So it's not far-fetched to say these models didn't do a very good job.

I saw one explanation was voters were unwilling to identify themselves as Trump supporters. How could a model incorporate effects like that?

One macro explanation I read is the rise of populism. The question then is how could a statistical model capture a macro trend like that?

Are these prediction models out there putting too much weight on data from polls and sentiment, not enough from where the country is standing in a 100 year view? I am quoting a friend's comments.

Answer 1

In short, polling is not always easy. This election may have been the hardest.

Any time we are trying to do statistical inference, a fundamental question is whether our sample is a good representation of the population of interest. A typical assumption that is required for many types of statistical inference is that of having our sample being a completely random sample from the population of interest (and often, we also need samples to be independent). If these assumptions hold true, we typically have good measures of our uncertainty based on statistical theory.

But we definitively do not have these assumptions holding true with polls! We have exactly 0 samples from our population of interest: actual votes cast at election day. In this case, we cannot make any sort of valid inference without further, untestable assumptions about the data. Or at least, untestable until after election day.

Do we completely give up and say "50%-50%!"? Typically, no. We can try to make what we believe are reasonable assumptions about how the votes will be cast. For example, maybe we want to believe that polls are unbiased estimates for the election day votes, plus some certain unbiased temporal noise (i.e., evolving public opinion as time passes). I'm not an expert on polling methods, but I believe this is the type of model 538 uses. And in 2012, it worked pretty well. So those assumptions were probably pretty reasonable. Unfortunately, there's no real way of evaluating those assumptions, outside strictly qualitative reasoning. For more discussion on a similar topic, see the topic of Non-Ignorable missingness.

My theory for why polls did so poorly in 2016: the polls were not unbiased estimates of voter day behavior. That is, I would guess that Trump supporters (and likely Brexit supporters as well) were much more distrustful of pollsters. Remember that Mr. Trump actively denounced polls. As such, I think Trump supporters were less likely to report their voting intentions to pollsters than supporters of his opponents. I would speculate that this caused an unforeseen heavy bias in the polls.

How could analysts have accounted for this when using the poll data? Based on the poll data alone, there is no real way to do this in a quantitative way. The poll data does not tell you anything about those who did not participate. However, one may be able to improve the polls in a qualitative way, by choosing more reasonable (but untestable) assumptions about the relation between polling data and election day behavior. This is non-trivial and the truly difficult part of being a good pollster (note: I am not a pollster). Also note that the results were very surprising to the pundits as well, so it's not like there were obvious signs that the assumptions were wildly off this time.

Polling can be hard.

Answer 2

There are a number of sources of polling error:

• You find some people hard to reach

  This is corrected by doing demographic analysis, then correcting for your sampling bias. If your demographic analysis doesn't reflect the things that make people hard to reach, this correction does not repair the damage.

• People lie

  You can use historical rates at which people lie to pollsters to influence your model. As an example, historically people state they are going to vote 3rd party far more than they actually do on election day. Your corrections can be wrong here.

  These lies can also mess up your other corrections; if they lie about voting in the last election, they may be counted as a likely voter even if they are not, for example.

• Only the people who vote end up counting

  Someone can have lots of support, but if their supporters don't show up on election day, it doesn't count. This is why we have registered voter, likely voter, etc models. If these models are wrong, things don't work.

• Polling costs money

  Doing polls is expensive, and if you don't expect (say) Michigan to flip you might not poll it very often. This can lead to surprised where a state you polled 3 weeks before the election looks nothing like that on election day.

• People change their minds

  Over minutes, hours, days, weeks or months, people change their minds. Polling about "what you would do now" doesn't help much if they change their minds before it counts. There are models that guess roughly the rate at which people change their minds based off historical polls.

• Herding

  If everyone else states that Hillary is +3 and you get a poll showing Hillary +11 or Donald +1, you might question it. You might do another pass and see if there is an analysis failure. You might even throw it out and do another poll. When you get a Hillary +2 or +4 poll, you might not do it. Massive outliers, even if the statistical model says it happens sometimes, can make you "look bad".

  A particularly crappy form of this happened on election day, where everyone who released a poll magically converged to the same value; they probably where outlier polls, but nobody wants to be the one who said (say) Hillary +11 the day before this election. Being wrong in a herd hurts you less.

• Expected sampling error

  If you have 1 million people and you ask 100 perfectly random people and half say "Apple" and half say "Orange", the expected error you'd get from sampling is +/- 10 or so, even if none of the above problems occur. This last bit is what polls describe as their margin of error. Polls rarely describe what the above correction factors could introduce as error.

---

Nate Silver at 538 was one of the few polling aggregators that used conservative (cautious) means to handle the possibility of the above kinds of errors. He factored in the possibility of systemic correlated errors in the polling models.

While other aggregators were predicting a 90%+ chance HC was elected, Nate Silver was stating 70%, because the polls were within "normal polling error" of a Donald victory.

This was a historical measure of model error, as opposed to raw statistical sampling error; what if the model and the corrections to the model were wrong?

---

People are still crunching the numbers. But, preliminary results indicate a big part of it was turnout models. Donald supporters showed up to the polls in larger numbers, and Hillary supporters in lesser numbers, than the polling models (and exit polls!) indicated.

Latino's voted more for Donald than expected. Blacks voted more for Donald than expected. (Most of both voted for Hillary). White women voted more for Donald than expected (more of them voted for Donald than Hillary, which was not expected).

Voter turnout was low in general. Democrats tend to win when there is high voter turnout, and Republicans when there is low.

Answer 3

This was mentioned in the comments on the accepted answer (hat-tip to Mehrdad), but I think it should be emphasized. 538 actually did this quite well this cycle^*.

538 is a polling aggregator that runs models against each state to try to predict the winner. Their final run gave Trump about a 30% chance of winning. That means if you ran three elections with data like this, you'd expect Team Red to win one of them. That isn't really that small of a chance. Its certainly a big enough one that I took precautions (eg: The Friday before I asked for Wednesday the 9th off at work, considering the likelihood of it being close enough to be a late night).

One thing 538 will tell you if you hang out there is that if polls are off, there's a good chance they will all be off in the same direction. This is for a couple of reasons.

• Likely voter models. Polls have to adjust for the the types of voters who will actually show up on election day. We have historical models, but this was obviously not your typical pair of candidates, so predicting based on past data was always going to be a bit of a crapshoot.
• Late election herding. Nobody wants to be the poll that blew the election the worst. So while they don't mind being an outlier in the middle of a campaign, at the end all the polls tend to tweak themselves so that they say the same thing. This is one of the things that was blamed for the polls being so egregiously off in Eric Cantor's surprise loss in 2014, and for the surprisingly close results of the 2014 Virginia Senate race as well.

^{* - 538 has now posted their own analysis. It mostly jibes with what is said above, but is worth reading if you want a lot more details.}

---

Now a bit of personal speculation. I was actually skeptical of 538's final % chances for its last 3 days. The reason goes back to that second bullet above. Let's take a look at the history of their model for this election (from their website)

(Sadly, the labels obscure it, but after this the curves diverged again for the last three days, out to more than a 70% chance for Clinton)

The pattern we see here is repeated divergence followed by decay back toward a Trump lead. The Clinton bubbles were all caused by events. The first was the conventions (normally there's a couple of days lag after an event for it to start showing up in the polling). The second seems to have been kicked off by the first debate, likely helped along by the TMZ tape. Then there's the third inflection point I've marked in the picture.

It happened on November 5, 3 days before the election. What event caused this? A couple days before that was another email-flareup, but that shouldn't have worked in Clinton's favor.

The best explanation I could come up with at the time was poll herding. It was only 3 days until the election, 2 days until the final polls, and pollsters would be starting to worry about their final results. The "conventional wisdom" this entire election (as evidenced by the betting models) was an easy Clinton win. So it seemed a distinct possibility that this wasn't a true inflection at all. If that were the case, the true curve from Nov 5 on was quite likely a continuation of this one towards convergence.

It would take a better mathematician than I to estimate the curve forward here without this suspicious final inflection point, but eyeballing it I think Nov 8 would have been near the crossover point. In front or behind depends on how much of that curve was actually real.

Now I can't say for sure this is what happened. There are other very plausible explanations (eg: Trump got his voters out far better than any pollster expected) But it was my theory for what was going on at the time, and it certainly proved predictive.

Answer 4

First it was Brexit, now the US election

Not really a first, e.g. the French presidential election, 2002 "led to serious discussions about polling techniques".

So it's not far-fetched to say these models didn't do a very good job.

Garbage in, garbage out.

I saw one explanation was voters were unwilling to identify themselves as Trump supporter. How could a model incorporate effects like that?

See response bias, and in particular social desirability bias. Other interesting reads: silent majority and Bradley effect.

Answer 5

The USC/LA Times poll has some accurate numbers. They predicted Trump to be in the lead. See The USC/L.A. Times poll saw what other surveys missed: A wave of Trump support

http://www.latimes.com/politics/la-na-pol-usc-latimes-poll-20161108-story.html

They had accurate numbers for 2012 as well.

You may want to review: http://graphics.latimes.com/usc-presidential-poll-dashboard/

And NY Times complained about their weighting: http://www.nytimes.com/2016/10/13/upshot/how-one-19-year-old-illinois-man-is-distorting-national-polling-averages.html

LA Times' response: http://www.latimes.com/politics/la-na-pol-daybreak-poll-questions-20161013-snap-story.html

Answer 6

No high ground claimed here. I work in a field (Monitoring and Evaluation) that is as rife with pseudo-science as any other social science you could name.

But here's the deal, the polling industry is supposedly in 'crisis' today because it got the US election predictions so wrong, social science in general has a replicability 'crisis' and back in the late 2000's we had a world financial 'crisis' because some practitioners believed that sub-prime mortgage derivatives were a valid form of financial data (if we give them the benefit of the doubt...).

And we all just blunder on regardless. Everyday I see the most questionable of researcher constructs used as data collection approaches, and therefore eventually used as data (everything from quasi-ordinal scales to utterly leading fixed response categories). Very few researchers even seem to realize they need to have a conceptual framework for such constructs before they can hope to understand their results. It is as if we have looked at market 'research' approaches and decided to adopt only the worst of their mistakes, with the addition of a little numerology on the side.

We want to be considered 'scientists', but the rigor is all a bit too hard to be bothered with, so we collect rubbish data and pray to the Loki-like god of statistics to magically over-ride the GIGO axiom.

But as the heavily quoted Mr Feynman points out:

“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong”.

There are better ways to handle the qualitative data which we are often stuck with, but they take a bit more work and those nice researcher constructs are often way easier to feed into SPSS. Convenience seems to trump science every time (no pun intended).

In short, if we do not start to get serious about raw data quality, I think we are just wasting everyone's time and money, including our own. So does anyone want to collaborate on a 'data quality initiative' in relation to social science methods (yes, there is plenty in the text books about such things, but no one seems to pay attention to that source after their exams).

Whoever has the most academic gravitas gets to be the lead! (It won't be me.)

Just to be clear about my answer here: I see serious fundamental issues with 'contrived' raw data types so often, that I would like to suggest a need to start at the beginning. So even before we worry about sampling or which tests to run on the data, we need to look at the validity/limitations of the data types we collect in relation to the models we are proposing. Otherwise the overall predictive model is incompletely defined.

Answer 7

Polls tend to have an error margin of 5% that you can't really get rid of, because it's not a random error, but a bias. Even if you average across many polls, it does not get much better. This has to do with misrepresented voter groups, lack of mobilization, inability to go to the vote on a workday, unwillingness to answer, unwillingness to answer right, spontaneous last-minute decisions, ... because this bias tends to be "correlated" across polls, you can't get rid of it with more polls; you also can't get rid of it with larger sample sizes; and you don't appear to be able to predict this bias either, because it changes too fast (and we elect presidents too rarely).

Due to the stupid winner-takes-all principle still present in almost all states, an error of 5% can cause very different results: Assume the polls always predicted 49-51, but the real result was 51-49 (so an error of just 2%), the outcome is 100% off; because of winner-takes-it-all.

If you look at individual states, most results are within the predicted error margins!

Probably the best you can do is sample this bias (+-5%), apply the winner-takes-all extremes, then aggregate the outcomes. This is probably similar to what 538 did; and in 30% of the samples Donald Trump won...

Answer 8

The reliance on data analysis had a huge impact in strategic campaign decisions, journalistic coverage, and ultimately in individual choices. What could possibly go wrong when the Clinton campaign's decisions were informed by no other than $\small 400,000$ daily simulations on the secret Ada algorithm?

In the end, it exposed a colossal failure of numerical analysis to make up for lack of knowledge of the subject matter. People were ashamed of themselves to explicitly embrace the winning candidate for obvious reasons.

The worst computer model could have gotten closer to the outcome if anybody had bothered to conduct a preliminary poll face to face, knocking on doors. Here is an example: the Trafalgar Group (no affiliation or knowledge other than what follows) had Trump leading in PA, FL, MI, GA, UT and NV (this latter state went ultimately blue) one day prior to the election. What was the magic?

a combination of survey respondents to both a standard ballot test and a ballot test guaging [sic] where respondent's neighbors stand. This addresses the underlying bias of traditional polling, wherein respondents are not wholly truthful about their position regarding highly controversial candidates.

Pretty low-tech, including the lack of spell-check, showing in numbers a lot about human nature. Here is the discrepancy in PA:

Historic Pennsylvania - so far from being perceived as the final straw in the Democratic defeat just hours prior to this closing realization at 1:40 am on November 9, 2016:

Answer 9

One of the reasons for poll inaccurracy in the US election, besides some people for whatever reason don´t say the truth is, that the "winner takes it all" effect makes predictions even less easier. A 1% difference in one state can lead to a complete shift of a state and influence the whole outcome very heavily. Hillary had more voters just like Al Gore vs Bush.

The Brexit referendum was not a normal election and therefore also harder to predict (No good historical data and everyone was like a first time voter on this matter). People who for decades vote for the same party stabilize predictions.

Answer 10

(Just answering this bit, as the other answers seem to have covered everything else.)

As late as 4 pm PST yesterday, the betting markets were still favoring Hillary 4 to 1. I take it that the betting markets, with real money on the line, should act as an ensemble of all the available prediction models out there.

No... but indirectly yes.

The betting markets are designed so the bookies make a profit whatever happens. E.g. say the current odds quoted were 1-4 on Hilary, and 3-1 on Trump. If the next ten people all bet \$10 on Hilary, then that \$100 taken in is going to cost them \$25 if Hilary wins. So they shorten Hilary to 1-5, and raise Trump to 4-1. More people now bet on Trump, and balance is restored. I.e. it is purely based on how people bet, not on the pundits or the prediction models.

But, of course, the customers of the bookies are looking at those polls, and listening to those pundits. They hear that Hilary is 3% ahead, a dead cert to win, and decide a quick way to make \$10 is to bet \$40 on her.

Indirectly the pundits and polls are moving the odds.

(Some people also notice all their friends at work are going to vote Trump, so make a bet on him; others notice all their Facebook friend's posts are pro-Hilary, so make a bet on her, so there is a bit of reality influencing them, in that way.)

Answer 11

It is not surprising that these efforts failed, when you consider the disparity between what information the models have access to and what information drives behavior at the polling booth. I'm speculating, but the models probably take into account:

• a variety of pre-election polling results
• historical state leanings (blue/red)
• historical results of prior elections with current state leanings/projections

But, pre-election polls are unreliable (we've seen constant failures in the past), states can flip, and there haven't been enough election cycles in our history to account for the multitude of situations that can, and do, arise.

Another complication is the confluence of the popular vote with the electoral college. As we saw in this election, the popular vote can be extremely close within a state, but once the state is won, all votes go to one candidate, which is why the map has so much red.

Answer 12

The polling models didn't consider how many Libertarians might switch from Johnson to Trump when it came to actual voting. The states which were won by a thin margin were won based on which percentage of the vote Johnson got. PA (which pushed Trump past 270 on the election night) gave only 2% to Johnson. NH (which went to Clinton) gave 4%+ to Johnson. Johnson was polling at 4%-5% the day before the election and he got roughly 3% on the day of the election.

So why did Libertarians, all of a sudden, switch on the day of the election? No one considered what was the central issue to Libertarian voters. They tend to view literal interpretation of the Constitution as canon. Most people who voted for Clinton did not think that her dismissiveness of the law was a high enough priority to consider. Certainly, not higher than everything which they didn't like about Trump.

Regardless of whether her legal troubles were important or not to others, they would be important to Libertarians. They would put a very high priority on keeping out of office someone who viewed legal compliance as optional, at best. So, for a large number of them, keeping Clinton out of office would become a higher priority than making a statement that Libertarian philosophy is a viable political philosophy.

Many of them may not have even liked Trump, but if they thought that he would be more respectful of the rule of law than Clinton would be, pragmatism would have won over principles for a lot of them and caused them to switch their vote when it came time to actually vote.

Answer 13

Polls are not historical trends. A Bayesian would inquire as to the historical trends. Since Abraham Lincoln, there has been a Republican party and a Democratic party holding the presidential office. The trend for party change 16 times since then from Wikipedia has the following cumulative mass function

where time in years to a change of presidential party is on the $x$-axis. After 8-years of a party in power, the odds are 68.75% that the voters vote for a change, just over 2 to 1. Moreover, since the 1860 election, Republicans have held the presidency 59% of the time versus 41% for Democrats.

What made journalists, the Democratic party, and the pollsters think that the odds were in favor of liberals winning was perhaps wishful thinking. Behavior may be predictable, within limits, but in this case the Democrats were wishing that people would not vote for a change, and from a historical perspective, it seems more likely there would be one than not.

Answer 14

I think poll results were extrapolated to the extent of the public assuming the voter demographics will be similar to poll taker demographics and would be a good representation of the whole population. For example, if 7 out of 10 minorities supported Hillary in the polls, and if that minority represents 30% of the US population, the majority of polls assumed 30% of voters will be represented by that minority and translated to that 21% gain for Hillary. In reality, white, middle-to-upper class males were better represented among the voters. Less than 50% of eligible people voted and this didn't translate into 50% off all genders, races, etc.

Or, polls assumed perfect randomization and based their models on that but in reality the voter data was biased toward older middle-to-upper class males.

Or, the polls didn't exactly assume perfect randomization but their extrapolation parameters underestimated the heterogeneity of voter demographics.

ETA: Polls of previous two elections performed better because of increased attention to voting by groups that aren't usually represented well.

Answer 15

HoraceT and CliffAB (sorry too long for comments) I’m afraid I have a lifetime of examples, which have also taught me that I need to be very careful with their explanation, if I wish to avoid offending people. So while I don’t want your indulgence, I do ask for your patience. Here goes:

To start with an extreme example, I once saw a proposed survey question that asked illiterate village farmers (South East Asia), to estimate their ‘economic rate of return’. Leaving the response options aside for now, we can hopefully all see that this a stupid thing to do, but consistently explaining why it is stupid is not so easy. Yes, we can simply say that it is stupid because the respondent will not understand the question and just dismiss it as a semantic issue. But this is really not good enough in a research context. The fact that this question was ever suggested implies that researchers have inherent variability on what they consider ‘stupid’. To address this more objectively, we must step back and transparently declare a relevant framework for decision making about such things. There are many such options, and I will use one that I sometimes find useful - but have no intent of defending here (I actively encourage anyone to think of others, as it means you are already starting down the road to better conceptualizations).

So, let’s transparently assume that we have two basic information types we can use in analyses: qualitative and quantitative. And that the two are related by a transformative process, such that all quantitative information started out as qualitative information but went through the following (oversimplified) steps:

1. Convention setting (e.g. we all decided that [regardless of how we individually perceive it], that we will all call the colour of a daytime open sky “blue”.)
2. Classification (e.g. we assess everything in a room by this convention and separate all items into ‘blue’ or ‘not blue’ categories)
3. Count (we count/detect the ‘quantity’ of blue things in the room)

Note that (under this model) without step 1, there is no such thing as a quality and if you don’t start with step 1, you can never generate a meaningful quantity.

Once stated, this all looks very obvious, but it is such sets of first principles that (I find) are most commonly overlooked and therefore result in ‘Garbage-In’.

So the ‘stupidity’ in the example above becomes very clearly definable as a failure to set a common convention between the researcher and the respondents. Of course this is an extreme example, but much more subtle mistakes can be equally garbage generating. Another example I have seen is a survey of farmers in rural Somalia, that asked “How has climate change affected your livelihood?” Again putting response options aside for the moment, I would suggest that even asking this of farmers in the Mid-West of the United States would constitute a serious failure to use a common convention between researcher and respondent (i.e. as to what is being measured as ‘climate change’).

Now let’s move on to response options. By allowing respondents to self-code responses from a set of multiple choice options or similar construct you are pushing this ‘convention’ issue into this aspect of questioning as well. This may be fine if we all stick to effectively ‘universal’ conventions in response categories (e.g. question: what town do you live in? response categories: list of all towns in research area [plus ‘not in this area’]). However, many researchers actually seem to take pride in the subtle nuancing of their questions and response categories to meet their needs. In the same survey that the ‘rate of economic return’ question appeared, the researcher also asked the respondents (poor villagers), to provide which economic sector they contributed to: with response categories of ‘production’, ‘service’, ‘manufacturing’ and ‘marketing’. Again a qualitative convention issue obviously arises here. However, because he made the responses mutually exclusive, such that respondents could only choose one option (because “it is easier to feed into SPSS that way”), and village farmers routinely produce crops, sell their labour, manufacture handicrafts and take everything to local markets themselves, this particular researcher did not just have a convention issue with his respondents, he had one with reality itself.

This is why old bores like myself will always recommend the more work intensive approach of applying coding to data post-collection - as at least you can adequately train coders in researcher-held conventions (and note that trying to impart such conventions to respondents in ‘survey instructions’ is a mug’s game –just trust me on this one for now). Also note also that if you accept the above ‘information model’ (which, again, I am not claiming you have to), it also shows why quasi-ordinal response scales have a bad reputation. It is not just the basic maths issues under the Steven’s convention (i.e. you need to define a meaningful origin even for ordinals, you can’t add and average them, etc. etc.), it is also that they have often never been through any transparently declared and logically consistent transformative process that would amount to ‘quantification’ (i.e. an extended version of the model used above that also encompasses generation of ‘ordinal quantities’ [-this is not hard to do]). Anyway, if it does not satisfy the requirements of being either qualitative or quantitative information, then the researcher is actually claiming to have discovered a new type of information outside the framework, and the onus is therefore on them to explain its fundamental conceptual basis fully (i.e. transparently define a new framework).

Finally let’s look at sampling issues (and I think this aligns with some of the other answers already here). For example, if a researcher wants to apply a convention of what constitutes a ‘liberal’ voter, they need to be sure that the demographic information they use to choose their sampling regime is consistent with this convention. This level is usually the easiest to identify and deal with as it is largely within researcher control and is most often the type of assumed qualitative convention that is transparently declared in research. This is also why it is the level usually discussed or critiqued, while the more fundamental issues go unaddressed.

So while pollers stick to questions like ‘who do you plan to vote for at this point in time?’, we are probably still ok, but many of them want to get much ‘fancier’ than this…