means and errors for election polls

Question

I'm trying to assess how to compute the errors associated with a numbers of likely voter polls of the Warnock Georgia Senate race. I have estimates of the margin of victory from 9 surveys of varying sample sizes. While these surveys likely have different approaches to their sample or weighting, I have no priors to weight some surveys higher than others except sample size. Also, to simplify things for my main question, you can assume there is no "undecided" category, so the results for the two candidate probabilities sum to 1. (This data is all from 538)

id estDemMargin (i.e., +2 means D= .51 & R =.49)     obs,
1    +2                                              713,
2    -1                                              550, 
3    +1                                              500,
4    -1                                              500, 
5    +4                                              857, 
6    +4                                             1680, 
7    +2                                             1011,
8    +3                                              578, 
9    +9                                              500

Were the polls off or not?

What are the relevant statistics to make this assessment? Is the estimated margin then +2.56 (mean of the reported margins) with an s.e. of 3.05 (s.e. of the reported margins) or s.e. = 0.96 (average of the individual survey s.e.'s). Or something else.

Both suggest the actual results (+2) is well within the CI, but there's obviously a much larger CI in the former case. Again, I am inserted in the fundamental question about how to compute the s.e. for a series of estimates less about the other vagaries of these polls (e.g., I don't care about incorporating Silver's grading of polling firms here)

Answer 1

I couldn't be sure of your notation, and I didn't see a final result there. So I went to '538' and got the data on the six polls currently listed for the Warnock race. I show below how I would combine the six polls to get an overall confidence interval.

For example, the first-listed poll of 857 likely voters showed 98% of them (840) choosing either Warnock or Loeffler, with 51% of them (437) for Warnock. [That's 52.02% for Warnock among subjects declaring a preference.]

Similar computations for all six polls are summarized in R below:

LV = c(857,500,500,550,1342,713)         # 6 polls: all subjects
Ln = round(Lv*c(.98,.98,.91,.97,.97,1))  #  subj for W or L
Lw = round(Lv*c(.51,.49,.46,.48,.49,.5)) #  subjects for W
Fw = Lw/Ln                               #  fraction for W
round(Fw, 4)
[1] 0.5202 0.5000 0.5055 0.4944 0.5054 0.4993
n = sum(Ln);  w = sum(Lw)
w; n; w/n
[1] 2190        # Total for W                                
[1] 4334        # Total for W or L
[1] 0.5053069   # Overall fraction for W

Altogether, there were 2190 responses for Warnock out of 4334 responses for either Warnock or Loeffler. Because we know of no reason to 'weight' the polls other than sample size, this seems a reasonable way to combine their findings. My method gives an overall $50.53\%$ of useful responses favoring Warnock.

A Jeffries 95% confidence interval $(0.4904, 0.5202)$ for the fraction of likely voters favoring Warnock is computed in R as follows:

qbeta(c(.025,.975), w+.5, (n-w)+.5)
[1] 0.4904225 0.5201842      # Jeffries Ci

Considered here as a frequentist confidence interval, this result is based on a Bayesian argument with the non-informative Bayesian prior $\mathsf{Beta}(0.5, 0.5).$ Taking into account the binomial likelihood based on $n = 4334$ useful responses, the ,Bayesian posterior distribution is $\mathsf{Beta}(2190.5, 2144.5),$ quantiles $0.025$ and $0.975$ of which give the confidence interval. [According to the Wikipedia page on binomial confidence intervals, this Jeffries interval estimate has good coverage properties when used as a frequentist confidence interval.]

Using the traditional (asymptotic) Wald 95% CI we get $(0.4904,0.5202).$ Specifically, this interval is of the form $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n},$ where $\hat p \approx 0.5053$ is the observed proportion in favor and $\sqrt{\hat p(1-\hat p)/n} \approx 0.0076$ is the (estimated) standard error.

On account of the relatively large sample sizes, it is not surprising that the two interval estimates match.

p.hat = 0.5053069;  n = 4334
p.hat + qnorm(c(.025,.975))*sqrt( p.hat*(1-p.hat) /n)
[1] 0.4904219 0.5201919     # Wald CI

According to the New York Times the percentage of the final vote for Warnock was 51.0%, so the point estimate from my 'combination' of polls is quite close to the final vote, and the CIs cover the final vote percentage.

Answer 2

Thanks. My question was more theoretical than computational. So you’re saying that the point estimate and error should be computed by pooling the “experiments,” rather than considering the polls 9 experiments. (I guess they are independent “draws” so this would be reasonable.)