12
$\begingroup$

I need help explaining, and citing basic statistics texts, papers or other references, why it is generally incorrect to use the margin of error (MOE) statistic reported in polling to naively declare a statistical tie.

An example: Candidate A leads Candidate B in a poll, $39 - 31$ percent, $4.5 \%$ margin-of-error for $500$ surveyed voters.

My friend reasons like so:

Because of the intricacies of statistical modeling, the margin of error means that A's true support could be as low as 34.5 percent and B's could be as high as 35.5 percent. Therefore, A and B are actually in a statistical dead heat.

All help appreciated in clearly articulating the flaw my friend's reasoning. I've tried to explain that it is incorrect to naively reject the hypothesis "A leads B" if $p_A-p_B < 2MOE$.

$\endgroup$
7
$\begingroup$

My first attempt at an answer was flawed (see below for the flawed answer). The reason it is flawed is that the margin of error (MOE) that is reported applies to a candidate's polling percentage but not to the difference of the percentages. My second attempt explicitly addresses the question posed by the OP a bit better.

Second Attempt

The OP's friend reasons as follows:

  1. Construct the confidence interval for Candidate A and Candidate B separately using the given MOE.
  2. If they overlap we have a statistical dead hear and if they do not then A is currently leading B.

The main issue here is that the first step is invalid. Constructing confidence intervals independently for the two candidates is not a valid step because the polling percentages for the two candidates are dependent random variables. In other words, a voter who decides not to vote for A may potentially decide to vote for B instead. Thus, the correct way to assess if the lead is significant or not is to construct a confidence interval for the difference. See the wiki as to how to compute the standard error for the difference of polling percentages under some assumptions.

Flawed answer below

In my opinion the 'correct' way to think of the polling result is as follows:

In a survey of 500 voters the chances that we will see a difference in lead as high as 8% is greater than 5%.

Whether you believe that 'A leads B' or 'A ties B' is then dependent on the extent to which you are willing to accept 5% as your cut-off criteria.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @Srikvant. Assume 5% is acceptable significance. I'm seeking a more precise answer, one which exposes the idea that "A leads B" is a new statistic, the difference of pA and pB, and that it's corresponding confidence interval is not simply 2*MOE. $\endgroup$ – somebody Aug 28 '10 at 23:39
4
$\begingroup$

It’s easier to explain in terms of standard deviations, rather than confidence intervals.

Your friend’s conclusion is basically correct under the simplest model where you have simple random sampling and two candidates. Now the sample proportions satisfy $p_A + p_B = 1$ so that $p_B = 1 - p_A$. Thus, $$Var(p_A - p_B) = Var(2 p_A - 1) = 4 Var(p_A)$$ and so $$SD(p_A - p_B) = 2 SD(p_A).$$ What makes this simple relationship possible is that $p_A$ and $p_B$ are perfectly negatively correlated, because in general $$Var(p_A - p_B) = Var(p_A) + Var(p_B) - 2 Cov(p_A, p_B).$$

Outside this simple model, if $p_A + p_B = 1$ does not hold in general, then you must take into account the correlation between $p_A$ and $p_B$ that is not included in the margin of error. It is possible for $SD(p_A - p_B) \ll 2 SD(p_A)$.

But all this nuance seems to indicate that the polling organizations should report the margin of error on the difference. Where’s Nate Silver?

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

Not only is that a bad way to term things but that's not even a statistical dead heat.

You don't use overlapping confidence intervals that way. If you really wanted to only say that Candidate A was going to win then Candidate A is definitely in the lead. The lead is 8% MOE 6.4%. The confidence interval of that subtraction score is not double the confidence interval of the individual scores. Which is implied by claiming the overlap of CIs (±MOE) around each estimate is a dead heat. Assuming equal N and variance, the MOE of the difference is sqrt(2) times 4.5. That's because finding the difference between the values would only double the variance (SD squared). The confidence interval is based on a sqrt of the variance therefore combining them is the average (4.5) * sqrt(2). Since the MOE of your 8% lead is approximately 6.4% then Candidate A is in the lead.

As an aside, MOE's are very conservative and based on the 50% choice value. The formula is sqrt(0.25/n) * 2. There is a formula for calculating standard errors of difference scores that we could use as well. We would apply that using the found values rather than the 50% cutoff and that still gives us a significant lead for Candidate A (7.5% MOE). I believe that, given the questioners comment, and the proximity of that cutoff to the hypothetical one selected, that that was probably what they were looking for.

Any introduction to both confidence intervals and to power would be helpful here. Even the wikipedia article on MOE looks pretty good.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy