Can you explain why statistical tie is not naively when p_1-p_2 < 2 MOE?

Question

I need help explaining, and citing basic statistics texts, papers or other references, why it is generally incorrect to use the margin of error (MOE) statistic reported in polling to naively declare a statistical tie.

An example: Candidate A leads Candidate B in a poll, 39 - 31 percent, 4.5 percent margin-of-error for 500 surveyed voters.

My friend reasons like so:

Because of the intricacies of statistical modeling, the margin of error means that A's true support could be as low as 34.5 percent and B's could be as high as 35.5 percent. Therefore, A and B are actually in a statistical dead heat.

All help appreciated in clearly articulating the flaw my friend's reasoning. I've tried to explain that it is incorrect to naively reject the hypothesis "A leads B" if $p_A-p_B < 2MOE$.

Answer 1

My first attempt at an answer was flawed (see below for the flawed answer). The reason it is flawed is that the margin of error (MOE) that is reported applies to a candidate's polling percentage but not to the difference of the percentages. My second attempt explicitly addresses the question posed by the OP a bit better.

Second Attempt

The OP's friend reasons as follows:

1. Construct the confidence interval for Candidate A and Candidate B separately using the given MOE.
2. If they overlap we have a statistical dead hear and if they do not then A is currently leading B.

The main issue here is that the first step is invalid. Constructing confidence intervals independently for the two candidates is not a valid step because the polling percentages for the two candidates are dependent random variables. In other words, a voter who decides not to vote for A may potentially decide to vote for B instead. Thus, the correct way to assess if the lead is significant or not is to construct a confidence interval for the difference. See the wiki as to how to compute the standard error for the difference of polling percentages under some assumptions.

Flawed answer below

In my opinion the 'correct' way to think of the polling result is as follows:

In a survey of 500 voters the chances that we will see a difference in lead as high as 8% is greater than 5%.

Whether you believe that 'A leads B' or 'A ties B' is then dependent on the extent to which you are willing to accept 5% as your cut-off criteria.

Answer 2

Not only is that a bad way to term things but that's not even a statistical dead heat.

The MOE is just one side of a confidence interval and you seem to be applying it correct in one aspect. Candidate B's score is unlikely to be higher than 35.5%. Since it's usually a 95% confidence interval then Candidate B's score will be higher than 35.5% less than 5% of the time. Not impossible, just usually considered unlikely. On the other hand, a score of say 34% also has a cutoff probability. It's just not reported. It's probably something lower than random chance, we just don't know what it is from the MOE alone. Therefore, it's completely incorrect to say that scores with the MOE cutoff are all just randomly equally likely. Which is the implication of the assertion of statistical dead heat.

Not only that, you don't use overlapping confidence intervals that way. If you really wanted to only say that Candidate A was going to win if the probability of Candidate B winning was less than 5% then Candidate A is definitely in the lead. The lead is 8%. The confidence interval of that subtraction score is not double the confidence interval of the individual scores but only sqrt(2) times those scores. That's because finding the difference between the values would only double the variance. The confidence interval is based on a sqrt of the variance therefore combining them is the average (4.5) * sqrt(2). Since the MOE of your 8% lead is approximately 6.4% then Candidate A is in the lead.

As an aside, MOE's are very conservative and based on the 50% choice value. The formula is sqrt(0.25/n) * 2. There is a formula for calculating standard errors of difference scores that we could use as well. We would apply that using the found values rather than the 50% cutoff and that still gives us a significant lead for Candidate A (7.5% MOE). I believe that, given the questioners comment, and the proximity of that cutoff to the hypothetical one selected, that that was probably what they were looking for. I like my answer better though because, while not the most correct mathematically in this instance, it is the generally correct way to compare data where confidence intervals are reported for individual values and gives a framework for thinking about why the CI doesn't double.

Any introduction to both confidence intervals and to power would be helpful here. Even the wikipedia article on MOE looks pretty good.

Answer 3

It’s easier to explain in terms of standard deviations, rather than confidence intervals.

Your friend’s conclusion is basically correct under the simplest model where you have simple random sampling and two candidates. Now the sample proportions satisfy $p_A + p_B = 1$ so that $p_B = 1 - p_A$. Thus, $$Var(p_A - p_B) = Var(2 p_A - 1) = 4 Var(p_A)$$ and so $$SD(p_A - p_B) = 2 SD(p_A).$$ What makes this simple relationship possible is that $p_A$ and $p_B$ are perfectly negatively correlated, because in general $$Var(p_A - p_B) = Var(p_A) + Var(p_B) - 2 Cov(p_A, p_B).$$

Outside this simple model, if $p_A + p_B = 1$ does not hold in general, then you must take into account the correlation between $p_A$ and $p_B$ that is not included in the margin of error. It is possible for $SD(p_A - p_B) \ll 2 SD(p_A)$.

But all this nuance seems to indicate that the polling organizations should report the margin of error on the difference. Where’s Nate Silver?

Answer 4

This answer is incorrect, as the true value A is not a random variable and hence P(A < some value) is meaningless. I tried to formalize the idea that overlapping confidence intervals doesn't mean that both samples are likely to be drawn from the same population. My bad.

---

WRONG ANSWER

More mathematically : Assuming the scores values are independent (which they aren't, but with the right setting it should be close), the chance that A has as low as 34.5 and B as high as 35.5 equals p(A<=34.5)*p(B>=35.5) = 0.000625.