How to interpret margin of error in a non-binary poll?

I know what margin of error means in a poll with two choices. For example, if a poll estimates two candidates are to have 45% and 55% of the vote respectively and the margin of error is 4%, then we can say with 95% certainty that the first candidate will receive between 41% and 49% of the vote. This means that the second candidate will have between 51% and 59% of the vote. We can say this because there are only two choices and the total must add up to 100%.

However, some polls sometimes measure the percentage expected votes for 3 or 4 candidates. In this case, if you have variance for the measurement of one candidate, how is that variance for one candidate's estimate distributed among two or more candidates?

• I do not think it means that. – Carl Jan 30 '17 at 6:51
• Where did I go wrong? Was looking at this for reference: pewresearch.org/fact-tank/2016/09/08/… – Pogomurphisk Jan 30 '17 at 8:04
• I think the real answer to this question is I really need to review my probability. I think random variables and distributions are a good place to start reviewing for me. – Pogomurphisk Jan 30 '17 at 18:40
• You would need to have different margins or error for every option. If only one is reported, I'd have to assume it's the largest margin of error, and so it's conservative for all other categories. – gammer Jan 31 '17 at 3:05
• @gammer Took your advice and deleted my answer. I do not know if there an adequate parametric probability theory that covers this. My suggestion is to bootstrap the data and get the 95% confidence intervals from that. And yes, that is an answer, but I will only post it if I can get people to upvote this comment. – Carl Jan 31 '17 at 4:54

Variance is not distributed among candidates. Each candidate can be considered in her own right, separately from all others, by recoding the responses as either for her or against her. Thus, the principles, theory, calculations, and interpretations of binary polls apply mutatis mutandis to polls with more than two options.

A model of poll results

The response for a particular candidate can be modeled as a binary random variable $X$. The poll itself is a sample of $X$, often enumerated $X_1, X_2, \ldots, X_n$. This means that each of the $X_i$ is a random variable like $X$, but all the $X_i$ are independent.

If we code a positive response as $1$ and a negative response as $0$, then the poll total $S = X_1+X_2+\cdots +X_n$ counts the people who favor this candidate. (Even though the final conclusions will be the same regardless of how the results are coded, this nice relationship is precisely why people use such 0-1 coding.) The percentages named in the question are realizations of the ratios $S/n$, which is the mean of the variables,

$$S/n = \frac{1}{n}\sum_{i=1}^n X_i = \bar X.$$

Suppose the expectation of $X$ is $p$: the true proportion of people in the population favoring the candidate. Although we never know $p$, we can still reason about it mathematically. In particular, $p$ determines the distribution of $X$: $X$ takes on the value $1$ with probability $p$ and otherwise has the value $0$ with probability $1-p$. $X$ is said to have a Bernoulli distribution with parameter $p$.

Standard errors

The variance of $X$ will determine the standard error of the poll. The next step is to compute this variance. The computation starts with the expectation. The expectation of $X$ is, by definition, its probability-weighted value $$E[X]=p(1) + (1-p)0= p.$$ Its variance, again by definition, is the expectation of $(X-E[X])^2$, computed as

$$\operatorname{Var}(X) = p(1 - p)^2 + (1-p)(0 - p)^2 = p(1-p).$$

It is useful to know this quantity varies between $0$ and $1/4$, attaining its maximum at $p=1/2$, and staying close to this maximum for $p\approx 1/2$. For instance, if $p=1/4$ or $p=3/4$, the variance of $1/4\times 3/4=3/16$ is still not much smaller than $1/4$.

The variance of $\bar X$ therefore equals

$$\operatorname{Var}(\bar X) = \operatorname{Var}\left(\frac{1}{n} S\right) = \frac{1}{n^2} \sum_{i=1}^n \operatorname{Var}(X) = \frac{1}{n^2}(np(1-p)) =\frac{p(1-p)}{n}.$$

The variance of the sum is the sum of the variances because the $X_i$ are independent.

The standard deviation of the poll mean is the square root of its variance. Since we don't know $p$, we estimate $p$ from the poll. This estimate $\hat p$ usually is taken to be the fraction of people favoring the candidate in the poll. Plugging this estimate into the variance formula and taking the square root gives the estimated standard deviation of the poll estimate, also known as its standard error:

$$\operatorname{SE}(\hat p) = \sqrt{\frac{\hat p(1-\hat p)}{n}}.$$

Because $\hat p(1-\hat p)$ will be close to (but slightly less than) $1/4$ when $\hat p$ is anywhere near $50\%$, we may conservatively overestimate the standard error even before taking the poll by using $1/4$ in the calculation. That is, the largest possible standard error of any response in a poll of $n$ observations is $\sqrt{1/(4n)}$.

Interpretation and Consequences

Even when the poll offers more than two options, if there is a possibility the response for one of those options might be anywhere near $1/2$, this conservative estimate still provides a good basis for designing the poll (that is, determining $n$, the number of people to poll). As an example, in a poll of $100$ people (such as a weekly tracking poll during a political campaign), $\sqrt{1/(4n)} = \sqrt{1/400} = 1/20 = 5\%.$ In a relatively expensive, well-designed poll perhaps $n=1000$ people will be reached, giving a (maximum possible) standard error of $\sqrt{1/4000} \approx 1.6\%$.

The 68-95-99.7 rule suggests about two-thirds of all polls will have estimates $\hat p$ within one standard error of the true values $p$, and that about $95\%$ of them will have estimates within two standard errors of the correct value. Thus, people often report two standard errors as the "margin of error" of a poll. The margin of error in a poll of $n=100$ therefore is going to be near $2\times 5\% = 10\%$, while the margin of error in a poll of $n=1000$ will be near $2\times 1.6\% \approx 3\%$. This is a pretty small value compared to responses near $50\%$ and so is often acceptable. The $3\%$ number is frequently quoted by authorities like the New York Times (which, in the linked article, also notes that the actual accuracy of political polls tends to be worse than this theoretical estimate).

If you wish, you may calculate standard errors yourself. All you need are the reported percentage $\hat p$ and the poll size $n$. This can be useful for obtaining nuanced interpretations of responses that are not near $50\%$. For instance, in a poll of $n=1000$ and a response of $\hat p = 10\% = 0.1$, you can easily compute a standard error of

$$\sqrt{\frac{\hat p(1-\hat p)}{n}} = \sqrt{\frac{0.1\times 0.9}{1000}} = \sqrt{90\times 10^{-6}} = \sqrt{90}\times 10^{-3}\approx 9.5\times 10^{-3} = 0.95\%.$$

Twice that would be around $2\%$, a little smaller than the nominal standard error of $3\%$ (as previously computed). However, the 68-95-99.7 rule is not as accurate for extremely low or high responses. Their distribution is skewed toward the middle rather than being symmetric. Thus, although the margin of error for a $10\%$ response in a poll of $1000$ indeed is $\pm 2\%$, you shouldn't be too surprised to discover (presumably later) that the true value was $p=13\%$, but you should be rather surprised to discover that it was only $p=8\%$. Rather than be concerned about such issues, it's often best just to go with the nominal (maximal) margin of error associated with the entire poll.

This entire analysis is valid when there are more than two possible responses to a poll question. The only complication is that the responses are correlated. This is obvious in the case of two options: whatever the proportion $\hat p$ might be for one option determines the proportion $1-\hat p$ for the other. But this has no effect whatsoever on the standard error calculation.

With some additional work, one can compute the correlations among such "multinomial" responses. This is rarely needed.