# Multiple Surveys and Confidence Interval

Hello and i please need some assistance. Every week I send out a survey request to 1,000 customers rating service from 1-10. Each week only 20% respond. Each week I report on the average. Because this is a sample, shouldn't I also report the margin of error? Follow up. Instead of reporting weekly, i sum all of the surveys for every week and report the results. This would change the margin of error calculation. Does that violate any principals? That feels incorrect as I am running weekly surveys.

• Is it possible that you have responses from some customers in several different weeks. At the extreme, could a customer respond every week? If so, you can't combine results from several weeks without violating the random sampling assumption. // The margin of error might be different from week to week because of different customers, various seasons, changing economic environment, etc. Commented Apr 22, 2022 at 17:34
• Yes, there is repeated surveying of individual customers. Fantastic point @BruceET. To confirm, margin of error reporting on a weekly basis is appropriate, right? I'm being told it isn't needed because all the weekly surveys add up to a large sample for the year. That doesn't feel accurate. Commented Apr 22, 2022 at 20:39

I agree that suitable reports of the (estimated) margin of error for relevant 95% CIs could be useful. The difficulty is that it is not clear which of several methods one would use for the margin of error.

First, the margin of error of a confidence interval for a binomial success probability $$p$$ based on $$n$$ observations depends on both $$p$$ and $$n.$$

Consider $$n = 100$$ Bernoulli trials with $$x = 5$$ successes. Then the estimated SE 0f estimate $$\hat p = 2/n$$ of $$p$$ is $$\sqrt{\hat p(1-\hat p)/n}.$$ Specifically, the margin of error is $$1.96\sqrt{\hat p(1-\hat p)/n} = 0.0427,$$ so that a 95% Wald confidence interval for $$p$$ is of the form $$\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}.$$

p.hat = 5/100; n = 100
1.96*sqrt(p.hat*(1-p.hat)/n)
0.04271721


Of course, if $$n = 100$$ remains the same and $$p$$ is larger, then one might expect a larger $$x$$ which would change the margin of error. Specifically, with, $$x = 50, n = 100$$ the estimated margin of error for the Wald interval would be $$0.098.$$

p.hat = 50/100; n = 100
1.96*sqrt(p.hat*(1-p.hat)/n)
[1] 0.098


So, you might be reporting a different Wald margin of error for each week, depending on the number of respondents and the number invited to take the poll.

Another complicating issue is that many different styles of confidence intervals are in common use. Some are based on normal approximations and some are not.

For an Agresti-Cooil 95% CI, which is more accurate than the Wald CI for smaller $$n,$$ one uses point estimate $$\tilde p = (x+2)/(n+4)$$ and margin of error $$1.96\sqrt{\tilde p(1-\tilde p)/(n+4)} = 0.0482.$$

p.est = 7/104
1.96*sqrt(p.est*(1-p.est)/(n+4))
[1] 0.04815495


Also, various styles of CIs are programmed into different binomial tests in R and in other statistical software. With $$x = 5, n = 100,$$ the R procedure prop.test (which essentially uses a normal approximation) would give the 95% CI $$(0.022, 0.112),$$ with implied margin of error $$0.045.$$ That is, any estimate of $$p$$ in this CI would not lead to rejecting $$H_0: p = 0.05.$$

prop.test(5, 100, p=.05)$conf.int [1] 0.02154368 0.11175047 attr(,"conf.level") [1] 0.95  By contrast, the R procedure binom.test uses an exact binomial distribution instead of a normal approximation. It's associated 95% CI is wider to make sure of 95% coverage probability for any value of $$p.$$ Because this interval is not centered at $$0.05,$$ it is difficult to say what margin of error is implied. binom.test(5, 100, p=.5)$conf.int
[1] 0.01643188 0.11283491
attr(,"conf.level")
[1] 0.95


In conclusion, I'm not saying that the task of providing an idea of the margin of error for the CIs you report is so impossibly difficult that you should abandon the idea. However, it does require you to understand the style of 95% CI you are using, and the statistical sophistication of the audience for the document in which such 'margins of error' are to be reported.

At the very least, you might start by rounding the reported CIs so that there is no false indication of accuracy. For example, report something like $$(0.02, 0.11),$$ instead of something like $$(0.0164, 0.1128).$$

• If you found this answer helpful, then please consider upvoting and/or accepting it. Commented Apr 25, 2022 at 13:53
• @BruceET, your thorough response is so appreciative. Commented Apr 25, 2022 at 14:03