Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Scenario:
A pastor has a congregation of 2,000 people.
He asks a sample of 10 random members whether they had a favourable or unfavourable experience relating to an event that was held.
8 out of 10 said unfavourable.
Assuming that all 2,000 members went to the event, how confident can he be that his random sample represented the population of 2,000 (e.g. that 1,600 would have reported it unfavourable)?
And what if only 1,000 attended? Or 300 only attended?
The only way to know whether a random sample is representative of the population is to compare features in both the sample and the entire population. Having only a sample and no other information about the population, one can not assess the representativeness.
This is where p-values and confidence intervals are useful. The confidence interval provides a range of plausible values for the unknown true population proportion based on the sample of size n=10.
You could use an asymptotic Wald confidence interval and consider incorporating a log or logit link function, i.e.
$$\text{exp}\Big( \text{log}\{\hat{p}\}\pm z_{1-\alpha/2}\Big[\sqrt{\hat{p}(1-\hat{p})/n}\Big]/\hat{p}\Big)$$
$$\text{or}$$
$$\text{logit}^{-1}\Big( \text{log}\Big\{\frac{\hat{p}}{1-\hat{p}}\Big\}\pm z_{1-\alpha/2}\Big[\sqrt{\hat{p}(1-\hat{p})/n}\Big]/[\hat{p}(1-\hat{p})]\Big).$$
The log link function is useful if your estimates are near 0 and your sample size is small. The logit link function is useful if your estimates are near 0 or near 1 and your sample size is small. You could also consider estimating the proportion that found the service favorable instead of unfavorable. Your estimate of $\hat{p}=2/10=0.20$ is not close to 0 by most standards, but the link functions may still help to improve the coverage of the confidence interval. For your estimate of $0.20$ they will shorten the lower confidence limit and lengthen the upper limit relative to a Wald interval using an identity link. Here log refers to the natural log with base $e$. For a $95\%$ confidence interval, $z_{1-\alpha/2}=1.96$.
| 95% confidence limits | |
|---|---|
| Wald with identity link | (-0.05, 0.45) |
| Wald with log link | (0.06, 0.69) |
| Wald with logit link | (0.05, 0.54) |
| Exact (Binomial CDF) | (0.02, 0.56) |
Of course one would not report a negative proportion so the lower limit of -0.05 from the Wald interval with identity link would be truncated to 0 (not inclusive). Each of these Wald intervals approximates the sampling distribution of the estimator in a slightly different way which explains the slightly different results. For inference on the proportion who found the service unfavorable you can subtract the point estimate and each confidence limit from 1. For reference I have also included exact confidence limits formed by inverting the CDF of a binomial distribution. Since the log link produces the widest interval one may prefer this method; however, the logit link produces results closest to the exact binomial method.
If your sample is representative then $20\%$ of the entire congregation found the service favorable. The upper confidence limit using the exact interval indicates that the true proportion of the congregation that found the service favorable could in fact be $56\%$ and your sample was an unrepresentative, uncommon $2.5\%$ event among similarly repeated experiments. Similarly, the lower confidence limit indicates that the true proportion of the congregation that found the service favorable could in fact be $2\%$ and your sample was an unrepresentative, uncommon $2.5\%$ event. Confidence intervals constructed in this manner will cover the unknown true proportion $95\%$ of the time in repeated experiments.
If the population (entire congregation) is small you may consider using a finite population correction often discussed in survey sampling, but even without the correction these limits are still useful and conservative.
