# z_star calculated via scipy can grow to infinty - Is this right?

After years of "abstinence" I dabble back into statistics of population proportions and have some trouble making sense of critical values $$z^\star$$ for high confidence levels:

The critical value $$z^\star$$ is needed for the calculation of confidence intervals. I compute $$z^\star$$ from the confidence level cl with the Python module scipy via

cdf = 0.5+cl/2
z_star = scipy.stats.norm.ppf(cdf)


This z_star (i.e. $$z^\star$$) can grow to infinity, because

scipy.stats.norm.ppf(1)


outputs inf.

On the other hand, I'd say the confidence interval

$$CI= \hat p \pm z^\star se(\hat p)$$

can reasonably not exceed $$[0\%,100\%]$$.

How do these things fit together?

• Is the scipy calculation an approximation with goes astray for confidence levels close to $$100\%$$?
• Or is my idea that confidence intervals not exceed $$[0\%,100\%]$$ not general enough?
• Or is something with my cdf calculation wrong?

I would love some pointers here!

• The confidence level has to be between 0 and 100, but the confidence interval can be any interval. – Dave Nov 11 '19 at 6:25
• Why can the CI be any interval? I'd say CI can be maximally [0%,100%]. Then it captures always the population proportion $p$ and therefor CI=[0%,100%] gives us a confidence level cl =1, doesn't it? – halloleo Nov 11 '19 at 6:40
• Oops, I hadn’t noticed that it’s a proportion. – Dave Nov 11 '19 at 11:46
• @Dave My bad. Added this crucial bit of information to the question. – halloleo Nov 12 '19 at 0:29

At the end there you're using a normal approximation to a binomial proportion to obtain a confidence interval. The normal is continuous and has an infinite range while the binomial proportion is discrete and has a finite range; if you go sufficiently far into the tail of the normal you can go outside the possible range of the binomial parameter, $$p$$.

As the Wikipedia article Binomial proportion confidence interval puts it:

A commonly used formula for a binomial confidence interval relies on approximating the distribution of error about a binomially-distributed observation, $$\hat p$$, with a normal distribution. This approximation is based on the central limit theorem and is unreliable when the sample size is small or the success probability is close to 0 or 1.

Here's an illustration of the problem: If that occurs, you're outside the region of values for $$p$$ and $$n$$ where the approximation is good for those percentiles. Indeed you would usually want to stay a fair way from the 0/1 endpoints (a fairly large distance in terms standard errors of the proportion, though the absolute distance from the end might be small in a large sample)

• Thanks for this. Your pointer helped me to read up on it on Wikipedia, etc. It makes sense now. – halloleo Nov 12 '19 at 0:37
• Sorry to lose your suggested edit: I was editing it myself at the same time and posting my edit meant yours was lost. I will put your edit in, though. (Edit: now done) – Glen_b -Reinstate Monica Nov 12 '19 at 1:05
• No problem. Your graph helps a lot! Thanks! – halloleo Nov 12 '19 at 1:13