8
$\begingroup$

The margin of error is driven by the size of the sample.

In a consultant's report (which is confidential at this stage), they collected responses from 10 store managers (out of a total of 200 store managers i.e. target population) and went on to make statements such as "only 20% of the respondents were happy with the sales performance of their stores".

The margin of error in this case is around 32%. Assuming a confidence level of 95%, the true response is between -12 and 52%.

How does one interpret this result (especially the negative part)?

It is wise to use margin of error for small sample sizes (e.g. sample of 15 out of a target population of 60)?

Improve this question
$\endgroup$
3
  • $\begingroup$ I alawys thought confidence interval and margin of error refered to the same concept! Is there a difference? $\endgroup$
    – Adhesh Josh
    Commented Jan 30, 2012 at 11:20
  • $\begingroup$ normally with samples of this size one would use a focus group methodology instead of a survey. $\endgroup$
    – Jamal Munshi
    Commented Nov 14, 2014 at 2:15
  • $\begingroup$ Related question here. $\endgroup$
    – Ben
    Commented Jan 22, 2021 at 1:07

3 Answers 3

Reset to default
17
$\begingroup$

Because it is not possible to have a percentage less than zero, the first interpretation is the response is that between 0% and 52% were happy. Just substitute 0 for the negative percentage.

The bigger question is why they are reporting results with such large margins of error. As a client, knowing that between 0% and 52% of respondents were happy is a pretty meaningless result. With such a small sample size, they are going to get those large margins of error for every single question in the survey. Ten people is just too small a sample size to get robust estimates.

For me, the bigger question is why they didn't use a qualitative method for this work.

Update based on comment below: increasing the margin of error to decrease the sample size is not a good way forward as you have seen with the margins of error you currently have. With a population of 60, you need to sample the entire population, i.e. undertaken a census, in order to get acceptably low margins of error. As for any survey, non-response bias will be a concern.

Recommendation: for the population of 60 either

  1. undertake a census if you want the results to be analysed statistically, or
  2. pick some key individuals in the population on the basis of their known attitudes or views, use qualitative interviews, and report themes (don't do any statistics, not even counts). For 10 people, qualitative interviewing is just as quick as doing a survey anyway.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Exactly my thoughts (Thanks, I can not put forward my case strongly). The difficulty for the consultant is that he must work with very samll target population, thus the sample sizes are low. A sample size of 100 gives a margin of error of about 10% (which I believe is acceptable). Should a margin of error be used for a sample size of less than 100 respondents. In addition, what would be the case if the target population is (say 60 and the sample is (say 15). I will be most grateful for addtional feedback on this matter. $\endgroup$
    – Adhesh Josh
    Commented Jan 28, 2012 at 14:08
  • $\begingroup$ When the population is small, the formulas for finite population correction factors show that sample sizes needed to achieve given levels of estimation accuracy are actually smaller than they would be with large populations. Thus, having a small population should not present any extra difficulties for the consultant. $\endgroup$
    – whuber ♦
    Commented Nov 14, 2014 at 3:53
6
$\begingroup$

The issue arises from the assumption of normality not quite being valid. The simple confidence interval typically reported assumes the sample means is normally distributed with a deviation equal to the standard error. This assumption is based on the central limit theorem, which requires large numbers.

It is possible to get exact Frequentist confidence intervals for small populations. For this case, with happy/not happy, the results are binary and the percent happy has a binomial distribution.

For this sample size, a more fun approach is a Bayesian approach -- what's the posterior distribution of beliefs in possible parameter values. The probability of being happy has a prior beta distribution. The formula is very simple, scroll down to Shrinkage Factors on this page. Beta Binomial Distribution on Wikipedia. It is easy to plot a nice graph in Excel showing the posterior distribution.

Improve this answer
$\endgroup$
3
  • 5
    $\begingroup$ Surely the simple confidence interval is based on the central limit theorem, not the strong law of large numbers? $\endgroup$
    – onestop
    Commented Jan 28, 2012 at 14:29
  • $\begingroup$ Let me give this a shot: Because this confidence interval is the result of a "large" set of numbers being expected to yield a normal distribution-- rather than to converge on some expected value. And if we were instead speculating about the results obtained from larger and larger samples, and how they would come closer and closer to the parameter, that would be an illustration of the strong law. Is that right? $\endgroup$
    – rolando2
    Commented Jan 28, 2012 at 16:33
  • $\begingroup$ The strong law would show that the parameter estimate would go to the true value of the parameter. But that says nothing about using a normal distribution to create a confidence interval. Without appealing to the CLT how do you know the sampling distribution isn't a Laplace distribution or something else? $\endgroup$
    – Dason
    Commented Jan 29, 2012 at 18:34
6
$\begingroup$

I agree with the statements already made here, but I have this tool to add:Newcombe's widely-cited proportion calculator. Using this tool, the confidence limits of 2/10 (the 20% case you mentioned) are {5.7%:51%}. Unless you gather more data, you're only really sure that half or less of the managers were happy.

Edit: linking to university sites is problematic, as the webpages are often restructured as faculty come and go.

Try this: vassarstats

Improve this answer
$\endgroup$
4
  • $\begingroup$ Excellent. Link seems down, try this one. http://faculty.vassar.edu/lowry/prop1.html $\endgroup$
    – prototype
    Commented Feb 9, 2012 at 2:06
  • $\begingroup$ note: updated link $\endgroup$
    – DocBuckets
    Commented May 22, 2012 at 23:17
  • $\begingroup$ Link is broken again $\endgroup$
    – Mysterious Otter
    Commented Jan 19, 2021 at 22:27
  • 1
    $\begingroup$ @zthomas.nc It looks like Newcombe is not at Cardiff anymore and Lowry is not at Vassar. This is the problem with university sources for websites. Try this: vassarstats.net $\endgroup$
    – DocBuckets
    Commented Jan 21, 2021 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.