2
$\begingroup$

Suppose that I made a survey with multiples questions where I have a target population of size nearly $N=100$ and a sample of size $n=8$. Is it possible to create any type of confidence interval for the overall results in this survey?

How is the overall margin of error reported in surveys? I've seen some material where the maximum margin of error of the survey is reported as

$$MOE=\frac{0.98}{\sqrt{n}}$$

In this case, I have a really small sample so the normal approximation doesn't seem correct to be used. Is there anything I can do about it?

$\endgroup$
4
  • 1
    $\begingroup$ Try using bootstrap to estimate the confidence intervals $\endgroup$
    – Allan
    Commented Aug 2, 2022 at 14:05
  • 2
    $\begingroup$ You may need to explain a little more of what statistic you want a confidence interval for. E.g. Is the result of this survey some kind of composite score ? Are you looking to estimate the mean score and confidence interval for this mean ? $\endgroup$ Commented Aug 8, 2022 at 20:18
  • $\begingroup$ @SalMangiafico I'm working with proportions and I will build intervals for each question of interest, but I also want to report a general confidence interval for the survey. I thought about using the Wilson interval with $\hat{p}=0.5$ since it would give a maximum width interval, but not sure if it's the right approach. $\endgroup$
    – R. Cowboy
    Commented Aug 11, 2022 at 15:37
  • $\begingroup$ To me, if you are looking at binomial proportions, using something like the Wilson method to calculate the confidence interval for the response for each question makes sense. However, because the sample size is small, you may want to use an exact method like Clopper-Pearson or Blaker. As far as I can tell, with n = 8, the Wilson interval will be more narrow... I don't know know how to calculate a confidence interval for the survey as a whole, unless the responses for each respondent can be summarized into an average or overall proportion. $\endgroup$ Commented Aug 11, 2022 at 16:20

2 Answers 2

3
+25
$\begingroup$

Use a standard one-sample confidence interval with adjustment for finite population

As a preliminary remark, I will note that a confidence interval applies to estimation of a specific unknown quantity in the analysis, so there is no such thing as a confidence interval for the "overall results" in a general sense. In the absence of contrary information, I will assume that our goal is to estimate the population mean of each quantity measured in your survey and this is what you mean by the "overall results" of the survey.

Assuming this is the case, the usual confidence interval to use would be the standard one-sample interval formed from the T-statistic, with adjustment for a finite population size. The formula for the confidence interval for the mean of a finite population (see e.g., O'Neill 2014, p. 286) is given by:

$$\text{CI}_N(1-\alpha) = \bigg[ \bar{x}_n \pm \sqrt{\frac{N-n}{N}} \cdot \frac{t_{n-1,\alpha/2}}{\sqrt{n}} \cdot s_n \bigg].$$

This interval can easily be computed using the CONF.mean function in the stat.extend package in R. You should note that the T-based confidence interval is usually based on use of the central limit theorem and in the present case your sample size is too small to use this effectively (though you'd be surprised at how rapidly some non-normal sampling distributions converge to normality when taking convolutions). If the underlying distribution of your data is far from the normal distribution then the confidence interval may be unreliable.

As a caveat on this recommendation, I will note here that some practitioners will passionately argue that the T-based confidence interval should not be used with a small sample size, since it is sensitive to the underlying distribution of your data (which you can't test with such a small sample). I tend to think that with a very small sample size you're going to get a pretty shitty inference no matter what you do, and bootstrapping $n=8$ data points is probably not going to be any better than the standard T-based confidence interval. If you examine examples of convergence to normality under convolutions you will also see that the CLT actually kicks in pretty strongly far earlier than the $n=30$ "rule of thumb" for most underlying distributions, so I tend to think that the CLT may be applicable even when the sample size is pretty small, so long as the underlying distribution is not hideous.

(Finally, I should note that if you are forming multiple confidence intervals for different survey variables, you should bear in mind the properties of multiple comparisons; even if you use a reasonably high confidence level, if you have a substantial number of intervals then some of them probably do not contain the true value of the estimated quantity.)

$\endgroup$
0
$\begingroup$

You could use Bootstrap confidence intervals. N=100 seems fine for me.

It works as follows:

  1. Draw 100 bootstrap resamples (i.e. redraw with replacement)
  2. then compute your statistics 100 times
  3. From these 100 statistic values, compute the 2.5 percentile and 97.5 percentile, both together estimate the 95% confidence region
$\endgroup$
2
  • 1
    $\begingroup$ Yes, but bootstrap which statistic ? $\endgroup$ Commented Aug 11, 2022 at 18:44
  • $\begingroup$ The value $N$ is the population size. The sample size is $n=8$. $\endgroup$
    – Ben
    Commented Feb 16, 2023 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.