Contradiction between confidence intervals and z-test

Question

I've got a problem with $z$-test and confidence intervals. The $z$-test rejects the null hypothesis (proportions are equal), but the confidence intervals are intersect.

Let us consider 2 companies. The proportion of smokers in company $A$ is $p(A)=0.22$. And the proportion of smokers in company $B$ is $p(B)=0.303$. The amount of employees in company $A$ is $403$. And the amount of employees in company $B$ is $404$.

So, we have the following data: \begin{align} p(A)&=0.22 &n(A)&=403 \\ p(B)&=0.303 &n(B)&=404 \end{align}

Now let's apply the $z$-test by the following formula for proportions (I took the formula from here):

$$z= \frac{p(B)-p(A)}{\sqrt{p(1-p)(\frac{1}{n(A)}+\frac{1}{n(b)})}}$$

Where $p$ is $$p=\frac{p(A)n(A)+p(B)n(B)}{n(A)+n(B)}$$

I made the following computation: $$p=\frac{0.22*403+0.303*404}{403+404}=0.2614$$

$$z= \frac{0.303-0.22}{\sqrt{0.2614(1-0.2614)(\frac{1}{403}+\frac{1}{404})}}=2.6828$$

So, $z$ more than $1.96$, it means that we reject the null hypothesis. The proportions $p(A)$ and $p(B)$ are not equal.

But then I've computed the 95% confidence intervals by the following formulas (I took them from here):

\begin{align} \newcommand{\low}{{\rm low}} \newcommand{\high}{{\rm high}} \low &= P-1.96 \sqrt{\frac{P(1-P)}{N}} \\ \high &= P+1.96 \sqrt{\frac{P(1-P)}{N}} \end{align} Where $P$ is proportion of smokers in some company and $N$ is amount of employees in this company.

I've computed the interval for company $A$: \begin{align} \low(A) &= p(A)-1.96 \sqrt{\frac{p(A)(1-p(A))}{n(A)}} = 0.22-1.96 \sqrt{\frac{0.22*0.78}{403}} = 0.1796 \\ \high(A) &= p(A)+1.96 \sqrt{\frac{p(A)(1-p(A))}{n(A)}} = 0.22+1.96 \sqrt{\frac{0.22*0.78}{403}} = 0.2604 \end{align} So, for $p(A)$ the CI is $(0.1796;0.2696)$.

And finally I've computed the interval for the company $B$: \begin{align} \low(B) &= p(B)-1.96 \sqrt{{p(B)(1-p(B))}{n(B)}} = 0.303-1.96 \sqrt{\frac{0.303 * 0.697}{404}} = 0.2581 \\ \high(B) &= p(B)+1.96 \sqrt{{p(B)(1-p(B))}{n(B)}} = 0.303+1.96 \sqrt{\frac{0.303 * 0.697}{404}} = 0.3477 \end{align} For $p(B)$ the confidence interval is $(0.2581; 0.3477)$.

And now we see that confidence intervals are intersect, but it contradicts the computations for $z$-test.

Finally I've used a web service for computing the $z$-test and the CI. The service drew the intersection of the CI's and rejected the null hypothesis of the $z$-test, too.

What is the cause of the contradiction?

Answer 1

You are doing the confidence interval portion of your analysis incorrectly.

Your null hypothesis is $H_{o}: p_1 = p_2$. This hypothesis is about the difference in proportions between company A and company B. However, for confidence intervals you computed two one-sample confidence intervals for company A and company B, and then looked for an overlap. This is not the same thing as a confidence interval for the difference in proportions. The droids formula you're looking is:

$ \hat{p}_{2} - \hat{p}_{1} \pm Z_{\frac{\alpha}{2}} \sqrt{ \frac{\hat{p}_{1}(1-\hat{p}_{1})}{n_1} + \frac{\hat{p}_{2}(1-\hat{p}_{2})}{n_2} } $

Substituting the appropriate values for a 95% confidence interval yields (0.022, 0.143), which agrees with the hypothesis test.

The CI for the difference is directly looking at what we are interested in: the difference between the proportions for companies A and B. Using two 1-sample CIs is an indirect means of comparison. If they didn't overlap, we could conclude that the two proportions did differ, but the opposite isn't true: their overlap tells us absolutely nothing about whether the two proportions differ. Two groups can show overlapping confidence intervals, and yet be significantly different when a two-sample procedure is used.

This link and this link discuss this issue a bit.

And two other things:

• Just a typo, but for future reference, hypothesis has a "y", not an "i".
• I'm not sure if this was a typo or a misunderstanding: The confidence level you used was 95%, not 5%. Your hypothesis test was conducted at the 5% level of significance, the confidence interval was at 95% confidence.

Answer 2

Confidence $\neq$ probability !

The apparent contradiction comes from an erroneous understanding of confidence intervals.

0.95 is the probability that the sampling procedure will provide a sample such that the confidence interval that can be computed with that sample will overlap the true value of the parameter. That .95 probability is therefore only relevant before the sample is drawn, not after. You cannot say that "the probability for the parameter to be in the confidence interval is 95%" once the interval is known (just like, once a coin has been tossed and turned up Heads, you cannot say it has probability 1/2 of being Tails, although that was true just before tossing it). This is why it is called a confidence interval and not a probable interval.

BTW, even if it were true that both intervals had a .95 chance of containing the parameter, as they are independent their intersection would have probability $.95^2$.