# How to determine confidence %?

Scenario:

A pastor has a congregation of 2,000 people.

He asks a sample of 10 random members whether they had a favourable or unfavourable experience relating to an event that was held.

8 out of 10 said unfavourable.

Assuming that all 2,000 members went to the event, how confident can he be that his random sample represented the population of 2,000 (e.g. that 1,600 would have reported it unfavourable)?

And what if only 1,000 attended? Or 300 only attended?

The only way to know whether a random sample is representative of the population is to compare features in both the sample and the entire population. Having only a sample and no other information about the population, one can not assess the representativeness.

This is where p-values and confidence intervals are useful. The confidence interval provides a range of plausible values for the unknown true population proportion based on the sample of size n=10.

You could use an asymptotic Wald confidence interval and consider incorporating a log or logit link function, i.e.

$$\text{exp}\Big( \text{log}\{\hat{p}\}\pm z_{1-\alpha/2}\Big[\sqrt{\hat{p}(1-\hat{p})/n}\Big]/\hat{p}\Big)$$

$$\text{or}$$

$$\text{logit}^{-1}\Big( \text{log}\Big\{\frac{\hat{p}}{1-\hat{p}}\Big\}\pm z_{1-\alpha/2}\Big[\sqrt{\hat{p}(1-\hat{p})/n}\Big]/[\hat{p}(1-\hat{p})]\Big).$$

The log link function is useful if your estimates are near 0 and your sample size is small. The logit link function is useful if your estimates are near 0 or near 1 and your sample size is small. You could also consider estimating the proportion that found the service favorable instead of unfavorable. Your estimate of $$\hat{p}=2/10=0.20$$ is not close to 0 by most standards, but the link functions may still help to improve the coverage of the confidence interval. For your estimate of $$0.20$$ they will shorten the lower confidence limit and lengthen the upper limit relative to a Wald interval using an identity link. Here log refers to the natural log with base $$e$$. For a $$95\%$$ confidence interval, $$z_{1-\alpha/2}=1.96$$.

95% confidence limits
Wald with identity link (-0.05, 0.45)
Wald with log link (0.06, 0.69)
Wald with logit link (0.05, 0.54)
Exact (Binomial CDF) (0.02, 0.56)

Of course one would not report a negative proportion so the lower limit of -0.05 from the Wald interval with identity link would be truncated to 0 (not inclusive). Each of these Wald intervals approximates the sampling distribution of the estimator in a slightly different way which explains the slightly different results. For inference on the proportion who found the service unfavorable you can subtract the point estimate and each confidence limit from 1. For reference I have also included exact confidence limits formed by inverting the CDF of a binomial distribution. Since the log link produces the widest interval one may prefer this method; however, the logit link produces results closest to the exact binomial method.

If your sample is representative then $$20\%$$ of the entire congregation found the service favorable. The upper confidence limit using the exact interval indicates that the true proportion of the congregation that found the service favorable could in fact be $$56\%$$ and your sample was an unrepresentative, uncommon $$2.5\%$$ event among similarly repeated experiments. Similarly, the lower confidence limit indicates that the true proportion of the congregation that found the service favorable could in fact be $$2\%$$ and your sample was an unrepresentative, uncommon $$2.5\%$$ event. Confidence intervals constructed in this manner will cover the unknown true proportion $$95\%$$ of the time in repeated experiments.

If the population (entire congregation) is small you may consider using a finite population correction often discussed in survey sampling, but even without the correction these limits are still useful and conservative.

• It's been years since I did maths-statistics, but am I correct in recalling one can try assuming a normal distribution bell curve and go from there? Aug 15, 2021 at 11:59
• Yes, all three intervals I provided use a normal bell curve approximation. The Wald with identity link uses the normal approximation to the binomial sampling distribution behind $\hat{p}$. The log link approximates this sampling distribution with a log-normal (equivalently it approximates $log(\hat{p})$ with a normal distribution. The logit link approximates the sampling distribution of the estimated log odds with a normal curve. Aug 15, 2021 at 12:05
• Thanks so much for your answer and comments. If you had to report back to him with a plain-English statement, can I ask what one might say? Aug 15, 2021 at 12:12
• I just included an interpretation of the upper confidence limit near the bottom of my answer. Aug 15, 2021 at 12:16