# Difference in hypothesis testing using p-value and confidence interval

I have a simple problem

The sample population is 384 people and $12$% of it have a disease. The hypothesis is that $5$% of the real population have a disease. So, $H_0: \mu = 0.05$ and $H_a: \mu \neq 0.05$

First I calculate the Z-statistic. $$z = \frac{\hat{p}-p}{\sqrt{p*(1-p)/384}} = \frac{0.12-0.05}{\sqrt{0.05*(1-0.05)/384}} = 6.3$$

Using the p-value to test the hypothesis (I used R 2*(1-pnorm(6.3))): $$2*Φ(-|6.3|)= 2.976457e^{-10}$$ which is less than 0.05 level of significance, thus, reject the Null.

Using the confidence interval with 0.05 significance level:

$$\hat{p} \pm z*\sqrt{\hat{p}(1-\hat{p})/384} = 0.12 \pm 6.3*\sqrt{0.12(1-0.12)/384} \\ \mbox{I get the following} \\ 0.0155 < 0.05 <0.224$$

So according to the confidence interval I cannot reject the Null. Is it I doing something wrong or is it sometimes the case that there is a disagreement between the two approaches? Granted there are cases in which the the CI and the p-value disagree which one should I go with? Thanks.

With large $N$, your test statistic will be distributed as a normal. Thus, we can refer to your test statistic as "$z$", and we can also use "$z$" to refer to the asymptotic sampling distribution. However, these are not the same $z$'s. When you form your $1-\alpha\%$ confidence interval, you need to multiply the standard error by $z_{1-\alpha/2}$ to get the right increment. You then add (subtract) the product from your observed percentage to get the confidence limits. For example, if you wanted a $95\%$ confidence interval, you would multiply your standard error by $z_{.975} = 1.96$. If you use this value, your confidence interval is:
\begin{align} \hat{p} \pm z_{1-\alpha/2}\times \sqrt{\frac{\hat{p}(1-\hat{p})}{N}} &= 0.12 \pm 1.96\times\sqrt{\frac{0.12(1-0.12)}{384}} \\[10pt] &= 0.0872 < 0.12 < 0.15 \end{align} which agrees with your hypothesis test.

• I just realized you welcomed me to this website three years ago when I asked my first question. Good to see you still active.
– Koba
Nov 20 '15 at 8:18
• You're welcome, @Koba. What question are you referring to? Nov 20 '15 at 16:22
• This one presumably. Nov 20 '15 at 18:25

I don't have rep to comment, but I see some flaws in that problem. First, 6.3 sigmas are way more than 0.95 C.L. Just apply quickly Chebyshev theorem. A confidence level says that "in N experiments, if at least 1 - p experiments you get the null hypothesis you can't reject it".

Then, a binomial distribution isn't symmetrical,but there are enough observations so it seems like symmetrical if you do an histogram.

Just to grasp the concept, I've been working on a couple of plots to illustrate why the $z$ statistic calculates the standard error expected around the population or theoretical proportion (from the perspective of the Null hypothesis so to speak), and then figures out how many of these standard errors fit into the distance from the theoretical population proportion to the sample proportion - in this case ~$\small6$ - as shown on this plot with six arrow separating the population from the sample proportions: And why, on the other hand the confidence interval does the same thing, but from the perspective of the possible alternative hypothesis, or in other words, from the sample mean: it is the proportion found in the sample that is used to calculate the standard error. This latter calculation is plotted with the confidence interval shown as diverging arrows away from the sample proportion, and covering two standard errors on either side (the confidence interval): In either case the conclusion is the same: rejecting $H_o$ in favor of $H_a: \,\mu \neq0.05$.

Code for illustrations here.

• The figures are good, but I feel that they don't even start to explain the OP's confusion. The core confusion is that even though sample proportion is 6 standard errors away from 0.5, the 95% confidence interval is not sample proportion plus minus 6 standard errors; it's only plus minus 2 standard errors. I feel like your second figure could show this explicitly. Nov 20 '15 at 18:12
• @amoeba Thank you for looking into it. If you have the time and inclination to give me concrete changes or additions you would consider, I will definitely reflect on them. Nov 20 '15 at 18:16
• Well, you show 6 SEs as 6 little arrows on your figure 1, so I thought you could show in your figure 2 that the width of the conf interval is given by only 2 of these little arrows. And then comment (in words) on this difference. Nov 20 '15 at 18:24
• @amoeba Thanks for the suggestion. I tried to change it including your tips. Nov 20 '15 at 19:22
• Good job! You have my +1. Nov 20 '15 at 21:38