I am totally new to stats and the field of confidence intervals. So this might be very trivial or even sound stupid. I would appreciate if you could help me understand or point me to some literature/text/blog that explains this better.

I see on various news sites like CNN, Fox news, Politico etc about their polls regarding the US Presidential race 2012. Each agency conducts some polls and reports some statistics of the form:

CNN: The popularity of Obama is X% with margin of error +/- x1%. Sample size 600. FOX: The popularity of Obama is Y% with margin of error +/- y1%. Sample size 800. XYZ: The popularity of Obama is Z% with margin of error +/- z1%. Sample size 300.

Here are my doubts:

  1. How do I decide which one to trust? Should it be based on the confidence interval, or should I assume that since Fox has a larger sample size, it's estimate is more reliable? Is there an implicit relationship between confidence itnervals and sample size such that specifying one obviates the need to specify the other?

  2. Can I determine standard deviation from confidence intervals? If so, is it valid always or valid only for certain distributions (like Gaussian)?

  3. Is there a way I can "merge" or "combine" the above three estimates and obtain my own estimate along with confidence intervals? What sample size should I claim in that case?

I have mentioned CNN/Fox only to better explain my example. I have no intention to start a Democrats vs Republicans debate here.

Please help me understand the issues that I have raised.


In addition to Peter's great answer, here are some answers to your specific questions:

  1. Who to trust will depend also on who is doing the poll and what effort they put into getting a good quality poll. A bigger sample size is not better if the sample is not representative, taking a huge poll, but only in one, non-swing state would not give very good results.

    There is a relationship between sample size and the width of the confidence interval, but other things also influence the width, such as how close the percentage is to 0, 1, or 0.5; what bias adjustments were used, how the sample was taken (clustering, stratification, etc.). The general rule is that the width of the confidence interval will be proportional to $\frac{1}{\sqrt{n}}$, so to halve the interval you need 4 times the sample size.

  2. If you know enough about how the sample was collected and what formula was used to compute the interval then you could solve for the standard deviation (you also need to know the confidence level being used, usually 0.05). But the formula is different for stratified vs. cluster samples. Also most polls look at percentages, so would use the binomial distribution.

  3. There are ways to combine the information, but you would generally need to know something about how the samples were collected, or be willing to make some form of assumptions about how the intervals were constructed. A Bayesian approach is one way.

  • 1
    $\begingroup$ +1. But is the outlook for #3 really so bleak or difficult? If I have a collection of independent estimates, each with its own margin of error, why can I not (at least roughly) combine them in the usual way (as a weighted mean, weighted inversely by squared MoEs) and combine their standard errors as well (using variance formulas)? It wouldn't be perfect, but it ought to be better than choosing one poll on which to rely, right? $\endgroup$ – whuber Oct 4 '12 at 20:00
  • $\begingroup$ Thanks Greg! I much appreciate your answers. You mentioned in your reply to question 3 that "A Bayesian approach is one way". Could you point me to some literature that gives more information on this? $\endgroup$ – Nik Oct 4 '12 at 20:02
  • $\begingroup$ @whuber: Thanks for your comment. That's what I had been thinking of doing. Do you think it is justified to combine these estimates in this fashion? May be not completely, but to a large extent? $\endgroup$ – Nik Oct 4 '12 at 20:11
  • 1
    $\begingroup$ @whuber, I did not mean to paint it as bleak, just to make sure that the poster was aware of and could live with the assumptions needed. $\endgroup$ – Greg Snow Oct 4 '12 at 21:17
  • $\begingroup$ @Nik, there are many tutorials on the web for Bayesian statistics. A simple approach (which would assume that the samples were all simple random samples, or that the survey design was such that the SRS assumption is not far off) would be to start with a beta prior, then use each poll with a binomial likelihood to update and get a new posterior. One nice thing about the Bayes approach is you can discount the effect of the previous studies if you don't want them to have as much influence as the most recent poll. $\endgroup$ – Greg Snow Oct 4 '12 at 21:23

This is a huge topic, but basically there are two issues:

1) Precision - this is determined by the sample size. Larger samples give more precise estimates with lower standard error and tighter confidence intervals

2) Bias - which, in statistics, doesn't necessarily have the negative connotations it does elsewhere. In polls, they try to get a random sample of XXXX (sometimes likely voters, sometimes registered voters). But, they don't. Some polls only use land lines. Different groups of people are more or less likely to answer. Different groups are more or less likely to just hang up.

So, all pollsters weight their responses. That is, they try to adjust their results to match known facts about voters. But they all do it a little differently. So, even with the same polling input data, they will give different numbers.

Who to trust? Well, if you look at Nate Silver's work on 538, he has ratings of how accurate pollsters were in previous elections. But that doesn't mean they will be equally accurate now.

  • $\begingroup$ Thanks Peter. So an estimate with lower margin of error is more 'precise'. Is there a way to also know how biased it is from just X% +/- x1% error margin? I guess that's not possible unless you know individual sample's preferences, right? $\endgroup$ – Nik Oct 4 '12 at 20:08
  • $\begingroup$ Yes, that's right. Of course, some pollsters have known biases (in one direction or another). Internal polls (run by one party or the other) are often biased. One way they can do this is by runnign several polls and only releasing those that are favorable. Then there's the whole issue of "push polls" in which questions about a candidate are prefaced by negative questions about him or her. $\endgroup$ – Peter Flom Oct 4 '12 at 20:20

This falls in the area of survey sampling. In principle the methods work because randomization is used. Here are the things that can differ in polls based on subjective decisions.

  1. Sampling frame. What group of voter should I draw my sample from?

  2. How do I handle the volatility of the undecided voter who may change his opinion on Obama vs Romney based on yesterday's poll or next weeks?

  3. Peter has touched on bias. The literary digest poll of 1936 was a disaster. It picked the Republican candidate over FDR because the sampling frame was based on random selection of telephone numbers. In 1936 only the upper middle class and the wealthy had phones. That group was dominated by Republicans who tend to vote for the Republican candidate. Roosevelt won by a landslide getting his votes from the poor and the middle class that tended to be very much a group of Democrats! That illustrate bias due to the subtly poor choice of a sampling frame.

  4. Survey sampling deals with finite populations. The population size is N. Say a simple random sample is drawn from that population and has size n. For simplicity assume only Obama and Romney are running. The proportion of votes Obama would get for this sampling frame is an average of binary variables (say 1 if the respondent picks Obama and 0 for Romney). The variance of the sample mean for this variable is [p(1-p)/n][N-n]/N where p is the true population proportin that would pick Obama. [N-n]/N is the finite population correction. in most polls N is much bigger than N and the correct can be ignored. Looking at p(1-p)/n we see the variance goes down with n. So if n is large the confidence interval at a given confidence level will get small. It is this variance (really its square root) that is used to get the margin of error that gets quoted.

Pollsters other survey samplers and statisticians at the US Census Bureau all have these statistical tools at their disposal and they do more complex and accurate methods (cluster random sample and stratified random sampling to mention a couple of methods).

When their modeling assumptions are valid the methods work remarkably well. Exit polling is a prime example. On election day you will see the networks accurately project winner in almost every state long before a near final count. That is because preelection day variability is gone. They know historically how people tended to vote and they can determine selected precincts in a way that avoids bias. The networks sometime differ. This can be due to a competion to pick the winner ahead of the others mentality. It also can in rare instances be because the vote is extremely close (e.g Presidential Election 2000 in Florida).

I hope this give you a clearer picture of what goes on. We no longer see gross mistakes like "Dewey defeats Truman" in 1948 or the Literary Digest fiasco of 1936. But statistics is not perfect and statisticians can never say that they are certain.

  • $\begingroup$ Thanks for the detailed explanation. This really helped! $\endgroup$ – Nik Oct 4 '12 at 20:03
  • $\begingroup$ We no longer see gross mistakes? So Clinton won in 2016, did she? I take your known unknowns and raise you a black swan. Like my daddy used to say, "It's what you don't know that kills you." $\endgroup$ – Carl Dec 20 '17 at 20:34
  • 1
    $\begingroup$ My answer had nothing to do with Clinton and the 2016 election which had many strange issues. (1) Russian intervention, (2) Clinton won the popular vote and (3) some Trump voters were reticent to admit that they would vote for Trump. Perhaps I should add that polls can be wrong when voter turnout is not what was expected. $\endgroup$ – Michael Chernick Dec 20 '17 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.