Confidence intervals vs sample size?

Question

I am totally new to stats and the field of confidence intervals. So this might be very trivial or even sound stupid. I would appreciate if you could help me understand or point me to some literature/text/blog that explains this better.

I see on various news sites like CNN, Fox news, Politico etc about their polls regarding the US Presidential race 2012. Each agency conducts some polls and reports some statistics of the form:

CNN: The popularity of Obama is X% with margin of error +/- x1%. Sample size 600. FOX: The popularity of Obama is Y% with margin of error +/- y1%. Sample size 800. XYZ: The popularity of Obama is Z% with margin of error +/- z1%. Sample size 300.

Here are my doubts:

1. How do I decide which one to trust? Should it be based on the confidence interval, or should I assume that since Fox has a larger sample size, it's estimate is more reliable? Is there an implicit relationship between confidence itnervals and sample size such that specifying one obviates the need to specify the other?

2. Can I determine standard deviation from confidence intervals? If so, is it valid always or valid only for certain distributions (like Gaussian)?

3. Is there a way I can "merge" or "combine" the above three estimates and obtain my own estimate along with confidence intervals? What sample size should I claim in that case?

I have mentioned CNN/Fox only to better explain my example. I have no intention to start a Democrats vs Republicans debate here.

Please help me understand the issues that I have raised.

Answer 1

In addition to Peter's great answer, here are some answers to your specific questions:

1. Who to trust will depend also on who is doing the poll and what effort they put into getting a good quality poll. A bigger sample size is not better if the sample is not representative, taking a huge poll, but only in one, non-swing state would not give very good results.

   There is a relationship between sample size and the width of the confidence interval, but other things also influence the width, such as how close the percentage is to 0, 1, or 0.5; what bias adjustments were used, how the sample was taken (clustering, stratification, etc.). The general rule is that the width of the confidence interval will be proportional to $\frac{1}{\sqrt{n}}$, so to halve the interval you need 4 times the sample size.

2. If you know enough about how the sample was collected and what formula was used to compute the interval then you could solve for the standard deviation (you also need to know the confidence level being used, usually 0.05). But the formula is different for stratified vs. cluster samples. Also most polls look at percentages, so would use the binomial distribution.

3. There are ways to combine the information, but you would generally need to know something about how the samples were collected, or be willing to make some form of assumptions about how the intervals were constructed. A Bayesian approach is one way.

Answer 2

This is a huge topic, but basically there are two issues:

1) Precision - this is determined by the sample size. Larger samples give more precise estimates with lower standard error and tighter confidence intervals

2) Bias - which, in statistics, doesn't necessarily have the negative connotations it does elsewhere. In polls, they try to get a random sample of XXXX (sometimes likely voters, sometimes registered voters). But, they don't. Some polls only use land lines. Different groups of people are more or less likely to answer. Different groups are more or less likely to just hang up.

So, all pollsters weight their responses. That is, they try to adjust their results to match known facts about voters. But they all do it a little differently. So, even with the same polling input data, they will give different numbers.

Who to trust? Well, if you look at Nate Silver's work on 538, he has ratings of how accurate pollsters were in previous elections. But that doesn't mean they will be equally accurate now.

Answer 3

This falls in the area of survey sampling. In principle, the methods work because randomization is used. Here are the things that can differ in polls based on subjective decisions.

1. Sampling frame. What group of voters should I draw my sample from?

2. How do I handle the volatility of the undecided voter who may change his opinion on Obama vs Romney based on yesterday's poll or next week?

3. Peter has touched on the bias. The literary digest poll of 1936 was a disaster. It picked the Republican candidate over FDR because the sampling frame was based on a random selection of telephone numbers. In 1936 only the upper middle class and the wealthy had phones. That group was dominated by Republicans who tend to vote for the Republican candidate. Roosevelt won by a landslide getting his votes from the poor and the middle class which tended to be very much a group of Democrats! That illustrates bias due to the subtly poor choice of a sampling frame.

4. Survey sampling deals with finite populations. The population size is N. Say a simple random sample is drawn from that population and has size n. For simplicity assume only Obama and Romney are running. The proportion of votes Obama would get for this sampling frame is an average of binary variables (say 1 if the respondent picks Obama and 0 for Romney). The variance of the sample mean for this variable is [p(1-p)/n][N-n]/N where p is the true population proportion that would pick Obama. [N-n]/N is the finite population correction. in most polls, N is much bigger than N and the correct can be ignored. Looking at p(1-p)/n we see the variance goes down with n. So if n is large the confidence interval at a given confidence level will get small. It is this variance (really its square root) that is used to get the margin of error that gets quoted.

Pollsters other survey samplers and statisticians at the US Census Bureau all have these statistical tools at their disposal and they do more complex and accurate methods (cluster random sample and stratified random sampling to mention a couple of methods).

When their modeling assumptions are valid the methods work remarkably well. Exit polling is a prime example. On election day you will see the networks accurately project winner in almost every state long before a near-final count. That is because preelection day variability is gone. They know historically how people tended to vote and they can determine selected precincts in a way that avoids bias. The networks sometimes differ. This can be due to a competition to pick the winner ahead of the others' mentality. It also can in rare instances be because the vote is extremely close (e.g Presidential Election 2000 in Florida).

I hope this gives you a clearer picture of what goes on. We no longer see gross mistakes like "Dewey defeats Truman" in 1948 or the Literary Digest fiasco of 1936. But statistics is not perfect and statisticians can never say that they are certain.