I'm a complete stats newbie, but I got into a discussion on the following article about polling accuracy. The author looks through the historic accuracy of UK pollsters, then finds the average prediction is 6% off the final result. From there he extrapolates that:

But if polls are missing election outcomes by 5 or 6 points on average, that means the margin of error (or 95 percent confidence interval) is very large indeed. Specifically, a 6-point average error in forecasting the final margin translates to a true margin of error of plus or minus 13 to 15 percentage points, depending on how you calculate it.

I went and looked into this as I wanted to check for myself and I don't see how you can calculate a Margin of Error from the actual error rates. It seems to me like this would be calculating the odds of a coin toss that has already happened and we know the results of, but the author is just a little bit more accomplished as a statistician than I am.

How do you calculate the MoE from these data? Is it possible or is he just editorialising?

  • $\begingroup$ The way your phrase your question sounds a bit bayesian, IMO, since you are trying to use past data in order to better inform you current estimates. Care to elaborate a bit more formally about your problem? $\endgroup$ May 22 '17 at 14:52

(not really an answer, too long for a comment)

The error margin given by polls is based solely on the number of people polled. It doesn't include errors like nonrandom sampling, people refusing to take the poll, people lying about what they believe, people changing their mind between poll time and election time, etc.

You could regard the true error margins (for a week before, two weeks before, etc) as probability distributions, and compute their mean/standard deviation. I think that's a reasonable way to look at it, especially if you consider the other factors (people not taking the poll, lying, etc) as also being random (ie, some polls having 10% liars, other polls having 20% liars).

  • $\begingroup$ Sorry to be dense, so I go and use the true error margins in the way you said and I could calculate standard deviation etc... but for margin of error, do I not just need sample size and population? I'm not sure how we get from those figures (actual poll accuracy) to that value (MoE). $\endgroup$ May 14 '17 at 17:08
  • $\begingroup$ For errors due to small sample size, you JUST need the sample size itself (it's something like 1/Sqrt[n], but don't quote me on that): you don't even need the population size unless the population is very small. For the "true error", yes, use the actual numbers given in the article. $\endgroup$
    – user1566
    May 14 '17 at 17:10

It seems to me like this would be calculating the odds of a coin toss that has already happened and we know the results of,

I think it's simpler than that. Nate Silver has a model for the polling error term, and he fit its parameters (in this case, one parameter, the standard deviation) based on one of its summaries (the average of past polling errors), much like the maximum likelihood estimation method for fitting a distribution parameters given a sample taken from it.

Say that the polling error $\epsilon$ is normally distributed with mean $\mu = 0$ and unknown standard deviation $\sigma$, i.e.

\begin{equation} \epsilon \sim \mathcal{N}(0, \sigma ^2) \end{equation}

Now $\epsilon$ can be negative or positive, but we would like to compute the average of all polling errors' magnitudes. We can describe this new random variable $M$ representing the magnitude of the polling error as

\begin{equation} M = \left| \epsilon \right| \end{equation}

We don't know how it is distributed, but we can describe its cumulative density function. The CDF of $M$ is such that

\begin{align} F_M(x) &= P(M<x) \\ &= P(\left| \epsilon \right| < x) \end{align}

Let's expand the norm above using two expressions:

\begin{align} F_M(x) &= P(-x < \epsilon < x) \\ &= P(\epsilon < x) - P(\epsilon < -x) \\ &= F_N(x, 0, \sigma) - F_N(-x, 0, \sigma) \end{align}

where $F_N(x, \mu, \sigma)$ is the CDF for the normal distribution with parameters $\mu$ and $\sigma$. We see above that $F_M(x)$ depends on the standard deviation of the polling error, so we'll call it $F_M(x, \sigma)$ from now on.

Given $M$'s CDF, the probability density function (PDF) of $M$, $f_M(x, \sigma)$, can be calculated by:

\begin{equation} f_M(x, \sigma) = \dfrac{d}{d\,x} F_M(x, \sigma) \end{equation}

Run it through the calculus machine and we get an expression for $f_M(x, \sigma)$:

\begin{equation} f_M(x, \sigma) = \sqrt{\dfrac{2}{\pi\, \sigma}} \cdot \exp \left(-\dfrac{x^2}{2\, \sigma ^2} \right) \qquad \qquad x \geqslant 0 \end{equation}

The average of the polling errors (which is what we went through all this for) is the mean of $M$; as such, it can be calculated using $f_M(x, \sigma)$:

\begin{equation} \dfrac{\mu_M}{\sigma} = \int _{0} ^{+\infty} x\, f_M(x, \sigma)\, \mathrm{d} x \end{equation}

The whole point is to estimate the polling error distribution parameter (its standard deviation) by using a summary of its sample (the average of polling errors). This last expression gives us just that, a relationship between the standard deviation of the polling error and the mean of its magnitude. Plugging in the numbers from Nate Silver's article and solving for $\sigma$, we get, for $\mu _M = 6\, p.p.$, $\sigma = 7.52\, p.p.$.

The true margin of error $\Delta$, as defined by Silver, is the half-amplitude of the $95\%$ confidence interval on the polling error:

\begin{align} P(-\Delta < \epsilon < \Delta) &= 95\% \\ P(\epsilon > \Delta) = P(\epsilon < -\Delta) &= 2.5\% \\ 1 - F_N(\Delta, 0, \sigma) &= 2.5\% \end{align}

Doing the math,

\begin{equation} \Delta \approx 1.96\, \sigma \end{equation}

Thus we then get a true margin of error of plus or minus $14.7\, p.p.$, as stated in the article.

I chose to represent the polling error as a normal random variable (as is common when doing OLS regression), but there are other equally good choices, which would yield different results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.