13
$\begingroup$

Polls out there (say, Gallup) sample some absurdly low number of people compared to the population size (e.g. maybe a thousand people out of hundreds of millions).

Now, to me, sampling a population as a means for estimating the population's statistics makes sense when you have a strong reason to believe the samples are representative of the population (or, similarly, of other samples).

For example, sampling obviously makes sense for medical studies, because we know a priori that humans all have quite similar genomes and that this factor makes their bodies behave similarly.
Note that this isn't some kind of loose coupling -- genome is a pretty damn strong determining factor.

However, I just don't understand what justifies using low sample sizes for things like political polls.

I could buy that maybe 80-90% of the people in any given neighborhood vote similarly for the president (due to similar socioeconomic/education backgrounds), but this hardly seems to justify the absurdly low number of samples. There is literally no compelling reason (at least to me) why 1000 random voters should behave like the 200 million other voters.

To me, you'd need at least like (say) 100× that amount. Why? I can think of a bunch of reasons, e.g.:

  1. There are ~22,000 precincts just in California. People grow up so differently in their economic and educational backgrounds that a poll of size 1000 seems laughably small. How can you summarize entire precincts with < 1 person on average?

  2. People can't generally change their bodies' responses to medicine, but they can change their opinions about politics just by thinking about it. The way I see it, there's no forcing factor akin to DNA in medicine when you're dealing with politics. At best I'd imagine there should be small pockets of correlation.

Yet somehow, polls like this seem to... work anyway? Or at least people seem to think they do?
But why should they? Maybe I just fundamentally don't understand sampling? Can someone explain?
I just can't take any of the polls I see seriously, but I feel like I'm in more or less alone in this...

$\endgroup$
  • 6
    $\begingroup$ "sampling obviously makes sense for medical studies, because we know a priori that humans all have quite similar genomes" I do not have medical background, but do really our DNA differs less than our political views? If so, why is it so hard to study genetics and why didn't we already have full understanding of it? I'd take a bet that if you take two random people then they will be more likely to have the same political views, then the same DNA. $\endgroup$ – Tim Nov 6 '16 at 23:06
  • 3
    $\begingroup$ @Tim: "I'd take a bet that if you take two random people then they will be more likely to have the same political views, then the same DNA." How much do you want to bet? google.com/search?q=dna+similarity+between+humans $\endgroup$ – user541686 Nov 6 '16 at 23:10
  • 2
    $\begingroup$ But isn't the 0.5% difference the most important thing when you make such comparisons? Also, we share 60% genes with flies, so I guess we could sample humans and flies exchangeably for medical research? Compared: in 2008 Obama got 53% votes in US presidential election. Moreover, I'd argue that when researching life-saving but potentially hazardous drug you should be more careful in sampling then when doing research about preference for using soap produced by company A vs B, or for opinion pools. $\endgroup$ – Tim Nov 6 '16 at 23:21
  • 2
    $\begingroup$ @user2338816: "It's convincing because it has been accurate historically" is less like math and more like science though. I'm completely willing to buy it on scientific grounds (because that's how science rolls), but not on purely mathematical (proof-based) grounds. $\endgroup$ – user541686 Nov 7 '16 at 9:25
  • 1
    $\begingroup$ I'd challenge the (cautious) claim that polls work for elections. I don't remember polls even being close to the actual results of elections where I'm from. There's simply too many factors you can't really account for - for example, with our ~60% attendance, you're almost as likely to sample someone who isn't going to vote than one who isn't. Participating in a survey is less effort than voting, and sometimes you even get paid for it. Some parties have much higher attendance than others (like the communist party). You need to cite the deviation as well as the "results" in any sampling. $\endgroup$ – Luaan Nov 7 '16 at 12:13
13
$\begingroup$

It seems like you're imagining a very simple sampling model.

The simplest model for sampling is called aptly Simple Random Sampling. You select a subset of the population (e.g., by dialing phone numbers at random) and ask whomever answers how they're voting. If 487 say Clinton, 463 say Trump, and the remainder give you some wacky answer, then the polling firm would report that 49% of voters prefer Clinton, while 46% prefer Trump. However, the polling firms do a lot more than this. A simple random sample gives equal weight to every data point. However, suppose your sample contains--by chance--600 men and 400 women, which clearly isn't representative of the population as a whole. If men as a group lean one way, while women lean the other, this will bias your result. However, since we have pretty good demographic statistics, you can weight* the responses by counting the women's responses a bit more and the men's a bit less, so that the weighted response represents the population better. Polling organizations have more complicated weighing models that can make a non-representative sample resemble a more representative one.

The idea of weighting the sampled responses is on pretty firm statistical ground, but there is some flexibility in choosing what factors contribute to the weights. Most pollsters do reweight based on demographic factors like gender, age, and race. Given this, you might think that party identification (Democratic, Republican, etc) should also be included, but it turns out that most polling firms do not use it in their weights: party (self)-identification is tangled up with the voter's choice in a way that makes it less useful.

Many polling outfits also report their results among "likely voters". In these, respondents are either selected or weighted based on the likelihood that they'll actually turn up to the polls. This model is undoubtedly data-driven too, but the precise choice of factors allows for some flexibility. For example, including interactions between the candidate and voter's race (or gender) wasn't even sensible until 2008 or 2016, but I suspect they have some predictive power now.

In theory, you could include all sorts of things as weighting factors: musical preference, eye color, etc. However, demographic factors are popular choices for weighting factors because:

  • Empirically, they correlate well with voter behavior. Obviously, there is no iron-clad law that 'forces' white men to be lean Republican, but over the last fifty years, they have tended to.
    • The population values are well known (e.g., from the census or Vital Records)

However, pollsters also see the same news everyone else does, and can adjust the weighting variables if necessary.

There are also some "fudge factors" that are sometimes invoked to explain poll results. For example, respondents sometimes are reluctant to give "socially-undesirable" answers. The Bradley Effect posits that white voters sometimes downplay their support for white candidates running against a minority to avoid appearing racist. It is named after Tom Bradley, an African-American gubernatorial candidate who narrowly lost the election despite leading comfortably in the polls.

Finally, you're completely correct that the very act of asking someone's opinion can change it. Polling firms try to write their questions in a neutral way. To avoid issues with the order of possible responses, the candidates' names might be listed in random order. Multiple versions of a question are also sometimes tested against each other. This effect can also be exploited for nefarious ends in a push poll, where the interviewer isn't actually interested in collecting responses but in influencing them. For example, a push poll might ask "Would you vote for [Candidate A] even if it was reported that he was a child molester?".


* You might also set explicit targets for your sample, like including 500 men and 500 women. This is called stratified sampling--the population is stratified into different groups, and each group is then sampled random. In practice, this isn't done very often for polls, because you'd need to stratify into a lot of exhaustive groups (e.g., college-educated men between 18-24 in Urban Texas).

$\endgroup$
  • 2
    $\begingroup$ I definitely understand they're not doing simple random sampling, but my question is about whether what they are doing is just good just because it happens that their assumptions are correct (i.e. a reasonable but subjective hunch), or whether the assumptions are also statistically justifiable. See my comment on the other answer here. $\endgroup$ – user541686 Nov 6 '16 at 23:34
  • 3
    $\begingroup$ Both, I think. Weighting the sample is the statistically correct thing to do, but there's....flexibility in deciding how choose what factors go into the weights. For example, race, gender, and education are all useful, but it turns out that party identification is often not (e.g., theguardian.com/commentisfree/2012/sep/27/…), probably because it's tied up with the voter's candidate choice. $\endgroup$ – Matt Krause Nov 7 '16 at 0:06
  • 1
    $\begingroup$ Similarly, the weights sometimes include an estimate of how likely the respondent is to vote: young people make a lot of noise, but don't always show up; the elderly rarely attend rallies but turn up reliably at the polls. This could be estimated from historical data (voter rolls are sometimes public), but I could imagine some places adjusted it for African-Americans in 2008 and for women in 2016. $\endgroup$ – Matt Krause Nov 7 '16 at 0:22
  • $\begingroup$ Thanks! Might be good to mention that "flexibility" in your answer too :) +1 $\endgroup$ – user541686 Nov 7 '16 at 0:31
10
$\begingroup$

There's a mathematical theorem called the "law of large numbers." Imagine that you want to determine the probability that a coin will come up heads. The "population" of coin flips is infinity - much larger than the 300,000,000+ people in the United States. But according to the Law of Large Numbers, the more coin flips you do, the more accurate your estimate will be.

The ideal poll: In the ideal poll, the pollsters would randomly choose names from the U.S. Census, they would find out where those people live, then they would go and knock on their door. If the person says they're planning on voting, the pollster asks who they're voting for and records their answer. Polling like this is mathematically guaranteed to work and the amount of error in your measurement for any given confidence level can be calculated easily.

Here's what the error means: Suppose that based on your poll, you got that there's a 52 percent chance Candidate Awesome McPerfect is going to win, with a 3% error with 98% confidence. That means that you can be 98% confident that the true portion of voters who favor candidate Awesome McPerfect is between 49% and 55%.

A Note on Error and Confidence For a given sample size, the more confident you are, the bigger your error will be. Think about it - you're 100% confident that the true proportion that support canditate Awesome is between 0% and 100% (most error possible), and you're 0% confident that the true proportion that supports canditate Awesome is exactly 52.0932840985028390984308% (zero error). More confidence means more error, less confidence means less error. However, the relationship between confidence and error is NOT linear! (See: https://en.wikipedia.org/wiki/Confidence_interval)

Polls in the real world: Because it's expensive to helicopter pollsters out to all parts of the country to knock on the doors of random people (although I'd love to see that happen; if you're a billionare and you see this, please consider funding this), polls in the real world are more complex. Lets look at one of the more common strategies - calling up random voters and asking them who they'd vote for. It's a good strategy, but it does have some well-aknowledged failings:

  1. People often choose not to answer the phone and respond to pollsters (ex. me)
  2. Some demographics are more likely to have a landline (ex. older voters)
  3. Some demographics are more likely to respond to pollsters (ex. older voters)

Because different demographics vote in different ways, pollsters have to do their best to control for the differences in their raw data (based on who decided to answer the phone) and the outcomes of actual elections. For example, if 10% of people who picked up the phone were hispanic, but 30% of voters in the last election were hispanic, then they're going to give three times the weight to hispanic voters in their poll. If 50% of people who answered the phone were older than 60, but only 30% of the people who voted in the last election were older than 60, they're going to give less weight to the older voters who responded. It's not perfect, but it can lead to some impressive feats of prediction (Nate Silver correctly predicted the results in each of the 50 states in the 2012 election using statistics, and he was correct in 49 out of the 50 states in the 2008 prediction).

A word of caution to the wise: Pollsters make the best predictions they can based on how things worked out in the past. Generally speaking, things work out about the same now as they did in the past, or at least the change is slow enough that the recent past (which they focus most on) will resemble the present. However, occasionally there are rapid shifts in the electorate and things go wrong. Maybe Trump voters are slightly less likely than your average voter to answer the phone, and weighting by demographics doesn't account for that. Or Maybe young people (who overwhelmingly support Hillary) are even more unlikely to answer the phone than the models predict, and the ones that do answer the phone are more likely to be republican. Or perhaps the opposite of both is true - we don't know. things like that are hidden variables that don't show up in commonly collected demographics.

We would know if we sent pollsters to knock on random doors (ahem, imaginary billionare reading this), since then we wouldn't have to weight things based on demographics, but until then, fingers crossed.

$\endgroup$
  • 3
    $\begingroup$ I appreciate the response, but it's a little bit elementary relative to the question I was trying to ask and my background (not sure if you noticed, but I'm not exactly new to the basics of probability/stats); I don't think the answer to my question here is as basic as yours. For example: an assumption for the classic law of large numbers is that we have random variables with identical distributions... but I fail to see a justification for it in a political context: why should the distribution you put on my vote and yours be the same at all? $\endgroup$ – user541686 Nov 7 '16 at 5:43
  • $\begingroup$ Also, I'm not even sure the law of large numbers justifies the thing you were trying to justify even if its assumptions are satisfied. The question is about sample sizes which the law of large numbers doesn't really address (at least not in the fashion you suggested); we need some notion of the variance or convergence rate here, not just the convergence of the mean at infinity. Maybe you meant to invoke the central limit theorem rather than the law of large numbers? (Though please see my previous comment since this is probably moot.) $\endgroup$ – user541686 Nov 7 '16 at 5:48
  • 2
    $\begingroup$ Distributions aren't applied to individual votes. Individual votes aren't random. They're applied to the voting behavior of the population as a whole. It's like drawing colored balls from an urn - each ball is predetermined to be red or blue, but you can have a probability of drawing each color and so you can construct a distribution for the likelihood of drawing a certain color of ball based on a sample of the balls in the urn $\endgroup$ – J. Antonio Perez Nov 7 '16 at 5:52
  • 1
    $\begingroup$ Lets look at something other than politics with people. Someone's favorite flavour of ice cream depends on just as many things as their political views. It could depend on the preferences of their friends, fond memories of their childhood, good or bad experiences at the ice-cream parlour. Perhaps they like one flavour because they got it on their first date with their wife or husband. Perhaps they dislike a flavour because it reminds them of their ex. But If I took a random poll of people in America, wouldn't you agree that I could judge the top favorite ice-cream flavours in America? $\endgroup$ – J. Antonio Perez Nov 7 '16 at 5:57
  • 1
    $\begingroup$ The "random variable" is which person is selected by the pollster to be asked their preference. The preference of an individual is not random; which individual the pollster selects is random. $\endgroup$ – J. Antonio Perez Nov 7 '16 at 5:58
7
$\begingroup$

Firstly, this is aside from your main points but it's worth mentioning. In the medical trial you could have 1000 people testing a drug which can be given to the 10000 people who are sick annually. You might look at that and think "That's being tested on 10% of the population", in fact the population isn't 10000 people, its all future patients so the population size is infinite. 1000 people isn't large compared to the infinite potential users of the drug but these kinds of studies work. It's not important whether you test 10%, 1% or 0.1% of the population; what's important is the absolute size of the sample not how big it is compared to the population.

Next, your main point is that there are so many confounding variables which can influence people's voting. You're treating the 22000 districts of California like 22000 variables but really they are just a handful of variables (income and education like you mentioned). You don't need a representative sample from every district, you just need enough samples to cover the variation due to income, education, ect.

If you have $k$ confounding variables (age, gender, education, ect) and they all have similar effects then the variance of the vote increases by about $k$ times. If you sample $n$ people then the variance of the sample average decreases by a factor of $n$. Therefore, if the variation from each confounding variable is $\sigma^2$ then your sample average from $n$ people with $k$ confounding variables will be $\frac{k\sigma^2}{n}$.

You can probably think of 10 or so confounding variables but the sample size is 1000 so $k$ is a lot smaller than $n$. Therefore the variance of the sample average is quite small.

Edit:

The above formula was assuming that each confounding variable is equally important. If we want to consider hundreds of things that can add variance to the results then this assumption isn't valid (e.g. maybe twitter users support one candidate more, but we know that twitter use isn't as important as gender).

We could list all the confounding variables in order of importance (e.g. gender, age, income, ... , twitter use, ...). Let's assume that each variable is only 90% as important as the previous one. Now if gender adds a variance equal to $\sigma^2$ then age adds a variance equal to $0.9 \sigma^2$ and income adds $0.9^2 \sigma^2 $. If we include an infinite number of confounding variables then the total variability is $\sum_{n=0}^{\infty} \sigma^2 0.9^n = 10 \sigma^2$.

With this type of consideration for minor variables we've ended up with a variance with 10 times the variability of gender alone. So with $n$ samples the variation in the sample average is $\frac{10\sigma^2}{n}$. Of course $0.9$ was chosen arbitrarily but this conveys a point about how these infinite number of minor variables should add up to something small

$\endgroup$
  • $\begingroup$ Thanks for the answer! Regarding the first point, I guess that's true, but my point there was that it doesn't even matter what the size of the human population is since you have a forcing factor (DNA, etc.) that would make the results quite similar for any sample. Regarding the second one, though: I can buy that there might be a few variables in practice, but the only way to justify that assumption mathematically and use it later is to actually sample a large number of people first and demonstrate it, right? Without that, the conclusion no longer seems statistically rigorous or justifiable. $\endgroup$ – user541686 Nov 6 '16 at 23:21
  • $\begingroup$ We have established by experiment that age, gender, income and a few others are key factors in people's voting pattern and we also know this just from general knowledge. You're right that there could be hundreds of other small factors which influence votes and in theory they could add up to something significant but our general knowledge tells us that they are unimportant. At this point the model is not rigorously justified but who is going to test minor factors like "Does being blonde make people vote for Clinton? Does wearing a wig make people vote for Trump?". $\endgroup$ – Hugh Nov 6 '16 at 23:33
  • $\begingroup$ "but who is going to test minor factors like [...]" -- but that's the issue here. If the answer is "because it's the best we can practically do/because it just so happens to work/because it's costly otherwise/etc.", that's a perfectly fine answer for the question of "Why aren't they polling 100,000 people?", but it's not really an answer to "How can 1,000 people be statistically justifiable?". That's why I'm asking this on Stats.SE as opposed to Politics.SE... I don't care if more samples is impractical; my question is why people think the current methods are statistically justified. $\endgroup$ – user541686 Nov 6 '16 at 23:39
  • $\begingroup$ The first couple of sentences in the last comment seem to be a reasonable answer though, if you're saying that that kind of study has been done on a large scale (~hundreds of thousands if not millions of people) and that that is the foundation of our assumptions. If so, I think they should be added to your answer since they address the crux of my question (ideally with some citation, though I can't really be too picky given it's a bit of a tangent and this isn't Politics.SE). $\endgroup$ – user541686 Nov 6 '16 at 23:43
  • $\begingroup$ You're right that testing minor factors is impractical but mathematically relevant. I've edited my answer to give some reasoning about why we aren't concerned about hundreds of small factors influencing the result. I'm sure you can find research done on the influence of the major factor. $\endgroup$ – Hugh Nov 7 '16 at 0:04
1
$\begingroup$

Perhaps someone can post a more enlightening answer as to why, but judging from the last two US elections, I'm forced to conclude exactly what I had suspected prior to the 2016 elections:

It appears our polling methodologies in fact do not work for elections.

There isn't yet a consensus on why this appears to be the case, but some hypotheses I've found have been the following:

  • Supporters of some candidates may be more likely to respond due to enthusiasm. [1, 2]
  • Supporters of some candidates may be less likely to respond due to distrust. [1, 2, 4]
  • We poll too often and compromise on sample size/quality, mistaking noise for signal. [1, 2]
  • Weightings that we do to correct for demographics biases may be inadequate. [3]

Links with more discussions on the topic:

  1. (NYMag) Should We Stop Paying Attention to Election Polls?

  2. (NYT) Why Did the Pre-Election Polls Get It So Wrong Again?

  3. (538) Trump Supporters Aren’t ‘Shy,’ But Polls Could Still Be Missing Some Of Them

  4. (Vox) Election results: Why the polls got it wrong

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.