# When sampling a given population many times the ratio of confidence intervals with the population proportion seems wrong

I wrote a Python script generating a population of yes/no votes with 50% of the votes set randomly to yes.

Then I take samples of 50 votes many times (say 10,100,100,10000 and 100000) and test for each sample whether its confidence interval for a given confidence level of 95% contains the population proportion.

I expected the ratio between the number of CIs which contain 𝑝 and the number of all generated CIs to become closer and closer to 0.95, but here is what I get:

  num of samples   ratio
---------------- ----------
10               1.0
100              0.99
1000             0.95
10000            0.9366
100000           0.93337
1000000          0.935186


This is looks like it gets closer and closer to 0.935 or so.

Is this still likely correct or is there rather a bug in my program?

### Some Details about my procedure:

I calculate the confidence interval CI from the confidence level cl like this

\begin{alignat*}{2} \text{CI}\; =\; \hat p\; &\pm\; &z^\star\:&\times\:\sqrt{\frac{\hat p \left(1 - \hat p\right)}{n}} \quad \text{with}\; z^\star \text{corresponding to a confidence level of}\; 95\% \end{alignat*}

or in Python code:

sigma_p_hat = math.sqrt(p_hat * (1 - p_hat) / n)
cdf = 0.5+cl/2
z_star = stats.norm.ppf(cdf)
E = z_star * sigma_p_hat
CI = pd.Interval(p_hat-E, p_hat+E, closed='both'

• What type of confidence interval did you use? Many of them only have approximate coverage, and the 'exact' types for proportions are approximate! – Michael Lew Nov 15 '19 at 1:27
• @MichaelLew Good point! I have added my method of calculating the confidence interval to the question. Re approximation: I thought the approximation only goes wrong for confidence levels very close to 100%. 95% is still quite a bit off. – halloleo Nov 15 '19 at 5:18
• What ratio calculation? – Glen_b -Reinstate Monica Nov 15 '19 at 7:39
• @Glen_b The ratio between CIs which contain $p$ and all generated CIs. Will add this to the question as well. – halloleo Nov 15 '19 at 8:20
• With a sample of 50, there are only 51 possible values of $\hat p$ and therefore only $51$ possible confidence intervals. That alone indicates it is unreasonable to expect the coverage to be exactly 95%. Are you therefore wondering why it is less than 95%? – whuber Nov 15 '19 at 20:34