Sensitivity analysis of election poll correlation

Question

I am interested in why the 2016 presidential election polls did a poor job in forecasting the result. One hypothesis I heard is that many of the polls conducted before the election were correlated, and a sensitivity analysis can be done to the datasets to figure this out.

My understanding of sensitivity analysis is the study of how errors are propagated and changed from the inputs to the output. But how can this be applied to election polls? The connection between the two is not obvious to me.

More specifically, given a series of poll datasets before the election, how can I do a sensitivity analysis to figure out how much impact does the correlation between polls have on the quality of the forecast of the election result?

Answer 1

A complete set of poll results can be found on the 538 website.

When the issue of correlation comes up, it's usually in the context of errors being correlated. Let's suppose Pennsylvania has a prediction of 50% +/- 3% and Ohio is 50% +/- 3% and Michigan is 50% +/- 3%. Then, one would think that the probability candidate A wins all 3 is .125 and the probability that candidate B wins all 3 is .125, if they were independent events.

But obviously they aren't independent. For one thing, late news events would show up on the news in all 3 states. For another, the weighting adjustments made (e.g. to adjust for the relative number of Democrats interviewed, the relative number of young people, etc.) will tend to be made similarly across states by the same pollster.

This introduces correlation among the errors, particularly the nonsampling errors.

If it were me, I'd start by looking at the Rasmussen state level polls, in part because Rasmussen is often accused of having a Republican bias, and compare these to some other polling organizations that also have frequent state level polls. Are the differences systematic? Are the errors relative to the final vote biased? Does the bias tend to follow any particular characteristic of the state (e.g. states which lost a lot of factory jobs)?

Answer 2

This is less of an answer and more a long comment.

My understanding of the big systemic error in polling

Who votes is a huge source of systematic error.

• Imagine we have two categories of people: Type A and Type B and that each make up 50% of the population of registered voters.

• Further imagine that we've polled literally everyone and we know:

  • 60% of type A voters intend to vote for candidate 1 and 40% vote for candidate 2
  • 35% of type B voters intend to vote for candidate 1 and 65% vote for candidate 2

Is there any uncertainty left? Yes! We don't know who actually will turn up to vote. Let $X$ be a random variable denoting the fraction of actual voters who are type A (hence $1-X$ are type B).

The share of the vote going for candidate 1 is therefore:

$$ .6X + .35(1 - X)$$

A hidden source of error in polls is their likely voter model. And this represents a systematic source of error. Sampling more and more people doesn't tell you much about who will actually go and vote. And it changes each election...

This article gets at the same concept from a slightly different angle.

Some thoughts on how to model this?

I'm not an expert on this at all! But I imagine some kind of sensitivity analysis you might do is:

• Map poll results -> Bernoulli random variable denoting chance of candidate winning
• Run a Monte Carlo simulation where the Bernoulli random variables have different levels of correlation to see how the distribution over the final result changes as a function of outcome correlation?

I remember seeing another question loosely on this topic recently?